3 questions:
1) What would a realistic scenario be where you'd want to use
:supersede ? I was thinking perhaps a situation where you want to
modify a file in place. So you do
(open filespec :direction :input) and then
(open filespec :direction :output :if-exists :supersede)
You read from the first stream , do the modifications you want
and write to the second stream. When you're finished you close
the first stream and then the second. Would this work ?
2) How would :supersede be implemented ? I was thinking write to
a temporary file and when the stream is closed move the temporary
file to its proper location. But how is the implementation
supposed to behave if you open a file for writing with :supersede
and before closing it you open it again for reading ? Are you
supposed to read what you wrote with :supersede or the original
content of the file ?
3) What would a "reasonable" behavior be for an implementation
which cannot do the right thing with :supersede ? I was thinking
the open should fail so that the programmer knows immediately
that the functionality is not available otherwise he may end up
irretrievably corrupting the file.
Spiros Bousbouras wrote:
> 1) What would a realistic scenario be where you'd want to use
> :supersede ? I was thinking perhaps a situation where you want to
> modify a file in place. So you do
> (open filespec :direction :input) and then
> (open filespec :direction :output :if-exists :supersede)
> You read from the first stream , do the modifications you want
> and write to the second stream. When you're finished you close
> the first stream and then the second. Would this work ?
I would expect that to be platform-independent. I'd expect it to fail
on MSWin filesystems and be implementation-dependent on POSIX filesystems.
> 2) How would :supersede be implemented ? I was thinking write to
> a temporary file and when the stream is closed move the temporary
> file to its proper location. But how is the implementation
> supposed to behave if you open a file for writing with :supersede
> and before closing it you open it again for reading ? Are you
> supposed to read what you wrote with :supersede or the original
> content of the file ?
As Pascal wrote, the ANSI spec is incomplete in this area. The
standard POSIX idiom for safely modifying a file is to update it
out-of-place as you describe. Open a new temp file, write to it,
(call fsync on ext4), close it, and copy over the old one. In the
interim, anyone opening the file will see the old version. If you
need concurrent reads to see the new version as it is being written,
you would need to use explicit synchronization and make changes
in-place. The exception being append operations; those are
synchronized by the OS.
Possible POSIX implementations of :supersede
- unlink() followed by creat()
- passing O_TRUNC to open()
The first would allow your example in question 1 to work properly.
The second would allow others to read the file as it is modified. If
this distinction is important, you probably have to step outsize the
ANSI spec.
- Daniel
On 12 Apr, 22:36, D Herring <········@at.tentpost.dot.com> wrote:
> Spiros Bousbouras wrote:
> > 1) What would a realistic scenario be where you'd want to use
> > :supersede ? I was thinking perhaps a situation where you want to
> > modify a file in place. So you do
> > (open filespec :direction :input) and then
> > (open filespec :direction :output :if-exists :supersede)
> > You read from the first stream , do the modifications you want
> > and write to the second stream. When you're finished you close
> > the first stream and then the second. Would this work ?
>
> I would expect that to be platform-independent. I'd expect it to fail
> on MSWin filesystems and be implementation-dependent on POSIX filesystems.
Do you mean platform-dependent ? Surely the functionality I'm
suggesting could be implemented on MS Windows. The question is
does the CL standard demand such a behavior ? If not then what's
the practical significance of "If possible, the implementation
should not destroy the old file until the new stream is closed."?
If you cannot read the old file after you open it with :supersede
and before the new stream is closed then for all practical purposes
it has been destroyed.
> > 2) How would :supersede be implemented ? I was thinking write to
> > a temporary file and when the stream is closed move the temporary
> > file to its proper location. But how is the implementation
> > supposed to behave if you open a file for writing with :supersede
> > and before closing it you open it again for reading ? Are you
> > supposed to read what you wrote with :supersede or the original
> > content of the file ?
>
> As Pascal wrote, the ANSI spec is incomplete in this area. The
> standard POSIX idiom for safely modifying a file is to update it
> out-of-place as you describe. Open a new temp file, write to it,
> (call fsync on ext4), close it, and copy over the old one. In the
> interim, anyone opening the file will see the old version. If you
> need concurrent reads to see the new version as it is being written,
> you would need to use explicit synchronization and make changes
> in-place. The exception being append operations; those are
> synchronized by the OS.
>
> Possible POSIX implementations of :supersede
> - unlink() followed by creat()
> - passing O_TRUNC to open()
>
> The first would allow your example in question 1 to work properly.
> The second would allow others to read the file as it is modified.
It seems to me that even with the first method others will be
able to read the file as it is modified. After all creat() is
equivalent to open(path, O_WRONLY|O_CREAT|O_TRUNC, mode)
> If
> this distinction is important, you probably have to step outsize the
> ANSI spec.
--
The notes I handle no better than many pianists. But the pauses
between the notes-ah, that is where the art resides.
Artur Schnabel
Spiros Bousbouras wrote:
> On 12 Apr, 22:36, D Herring <········@at.tentpost.dot.com> wrote:
>> Spiros Bousbouras wrote:
>>> 1) What would a realistic scenario be where you'd want to use
>>> :supersede ? I was thinking perhaps a situation where you want to
>>> modify a file in place. So you do
>>> (open filespec :direction :input) and then
>>> (open filespec :direction :output :if-exists :supersede)
>>> You read from the first stream , do the modifications you want
>>> and write to the second stream. When you're finished you close
>>> the first stream and then the second. Would this work ?
>> I would expect that to be platform-independent. I'd expect it to fail
>> on MSWin filesystems and be implementation-dependent on POSIX filesystems.
>
> Do you mean platform-dependent ? Surely the functionality I'm
> suggesting could be implemented on MS Windows. The question is
> does the CL standard demand such a behavior ? If not then what's
> the practical significance of "If possible, the implementation
> should not destroy the old file until the new stream is closed."?
> If you cannot read the old file after you open it with :supersede
> and before the new stream is closed then for all practical purposes
> it has been destroyed.
Yes, I meant platform-dependent. MSWin has locking semantics such
that you frequently can't delete a file when an application is reading
it. Unix filesystems generally don't care; when an open file is
deleted/unlinked, it will be garbage-collected when everything closes it.
If you O_TRUNC an open file before you have read it, there won't be
much left to read.
- Daniel
Spiros Bousbouras <······@gmail.com> writes:
> 3 questions:
>
> 1) What would a realistic scenario be where you'd want to use
> :supersede ?
Almost in all cases you want that. This is the most portable option
too, it seems.
> I was thinking perhaps a situation where you want to
> modify a file in place. So you do
> (open filespec :direction :input) and then
> (open filespec :direction :output :if-exists :supersede)
> You read from the first stream , do the modifications you want
> and write to the second stream. When you're finished you close
> the first stream and then the second. Would this work ?
You cannot open the same file twice at the same time.
> 2) How would :supersede be implemented ?
This is implementation dependant. But mostly, it does something right.
> I was thinking write to
> a temporary file and when the stream is closed move the temporary
> file to its proper location. But how is the implementation
> supposed to behave if you open a file for writing with :supersede
> and before closing it you open it again for reading ? Are you
> supposed to read what you wrote with :supersede or the original
> content of the file ?
No, it won't work. You will have to read the data before overwritting
it.
But you could also open the file :direction :io :if-exists :append
and _modify_ it, reading and writting at will.
> 3) What would a "reasonable" behavior be for an implementation
> which cannot do the right thing with :supersede ? I was thinking
> the open should fail so that the programmer knows immediately
> that the functionality is not available otherwise he may end up
> irretrievably corrupting the file.
CLHS says:
An implementation is required to recognize all of the open keyword
options and to do something reasonable in the context of the host
operating system. For example, if a file system does not support
distinct file versions and does not distinguish the notions of
deletion and expunging, :new-version might be treated the same as
:rename or :supersede, and :rename-and-delete might be treated the
same as :supersede.
So the behavior would have to be host OS dependant.
However, there seems that there is some leeway allowing CL
implementation to "interpret" the host environment, so we can get
different implementations on the same OS implementing different
behaviors for platform specific features.
In short, it's a mess.
For example, for your purpose above, :rename-and-delete could be a
better option. But see what happens in some implementations:
C/USER[4]> (defvar *i* (open "/tmp/messages.txt" :direction :input))
*I*
C/USER[5]> (defvar *o* (open "/tmp/messages.txt" :direction :output :if-exists :rename-and-delete))
** - Continuable Error
OPEN: #<INPUT BUFFERED FILE-STREAM CHARACTER #P"/tmp/messages.txt" @1> already
points to file "/private/tmp/messages.txt", opening the file again for
:OUTPUT may produce unexpected results
If you continue (by typing 'continue'): Open the file anyway
The following restarts are also available:
ABORT :R1 Abort main loop
C/Break 1 USER[6]> continue
*** - OPEN: Cannot rename file #P"/private/tmp/messages.txt" since there is a
file stream open to it
The following restarts are available:
ABORT :R1 Abort main loop
C/Break 1 USER[7]> abort
C/USER[9]>
(The culprit here is clisp, but you find similar quirks in other
implementations).
We would need a substandard indicating how pathname, file and stream
operations should be implemented on POSIX systems. A CDR could be
written, and implementations could be nudged toward it.
--
__Pascal Bourguignon__
http://www.informatimago.com
Pascal J. Bourguignon <···@informatimago.com> wrote:
+---------------
| Spiros Bousbouras <······@gmail.com> writes:
| > I was thinking perhaps a situation where you want to
| > modify a file in place. So you do
| > (open filespec :direction :input) and then
| > (open filespec :direction :output :if-exists :supersede)
| > You read from the first stream , do the modifications you want
| > and write to the second stream. When you're finished you close
| > the first stream and then the second. Would this work ?
|
| You cannot open the same file twice at the same time.
+---------------
I can't find any restriction against this in the CLHS "Function OPEN"
page. And, indeed, neither CMUCL nor CLISP seems to complain:
cmu> (with-open-file (s0 "foo1")
(with-open-file (s1 "foo1")
(with-open-file (s2 "foo1")
(mapcar #'read-line (list s0 s1 s2)))))
("Hello, there!" "Hello, there!" "Hello, there!")
cmu>
[1]> (with-open-file (s0 "foo1")
(with-open-file (s1 "foo1")
(with-open-file (s2 "foo1")
(mapcar #'read-line (list s0 s1 s2)))))
("Hello, there!" "Hello, there!" "Hello, there!")
[2]>
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> Pascal J. Bourguignon <···@informatimago.com> wrote:
> +---------------
> | Spiros Bousbouras <······@gmail.com> writes:
> | > I was thinking perhaps a situation where you want to
> | > modify a file in place. So you do
> | > (open filespec :direction :input) and then
> | > (open filespec :direction :output :if-exists :supersede)
> | > You read from the first stream , do the modifications you want
> | > and write to the second stream. When you're finished you close
> | > the first stream and then the second. Would this work ?
> |
> | You cannot open the same file twice at the same time.
> +---------------
>
> I can't find any restriction against this in the CLHS "Function OPEN"
> page. And, indeed, neither CMUCL nor CLISP seems to complain:
What more can I do? Didn't I paste a dribble of clisp complaining, in
by previous post that you savagely cut out instead of quoting
meaningfully.
--
__Pascal Bourguignon__
http://www.informatimago.com
On 13 Apr, 12:22, ····@informatimago.com (Pascal J. Bourguignon)
wrote:
> ····@rpw3.org (Rob Warnock) writes:
> > Pascal J. Bourguignon <····@informatimago.com> wrote:
> > +---------------
> > | Spiros Bousbouras <······@gmail.com> writes:
> > | > I was thinking perhaps a situation where you want to
> > | > modify a file in place. So you do
> > | > (open filespec :direction :input) and then
> > | > (open filespec :direction :output :if-exists :supersede)
> > | > You read from the first stream , do the modifications you want
> > | > and write to the second stream. When you're finished you close
> > | > the first stream and then the second. Would this work ?
> > |
> > | You cannot open the same file twice at the same time.
> > +---------------
>
> > I can't find any restriction against this in the CLHS "Function OPEN"
> > page. And, indeed, neither CMUCL nor CLISP seems to complain:
>
> What more can I do? Didn't I paste a dribble of clisp complaining, in
> by previous post that you savagely cut out instead of quoting
> meaningfully.
Perhaps explain what you mean by "you cannot do <whatever>". I
take it to mean that the standard makes it undefined behavior or
that it won't produce a predictable result in a correctly
functioning implementation. The fact that an implementation gives
you a warning is not enough to conclude any of the above. It
could be a limitation or bug of the particular implementation.
And even there , for all we know , if we ignore the warning
and go ahead things might work fine.
--
Even real life has plot holes.
On 12 Apr, 21:58, ····@informatimago.com (Pascal J. Bourguignon)
wrote:
> Spiros Bousbouras <······@gmail.com> writes:
> > 3 questions:
>
> > 1) What would a realistic scenario be where you'd want to use
> > :supersede ?
>
> Almost in all cases you want that. This is the most portable option
> too, it seems.
If you want to overwrite it why not use :overwrite instead of
:supersede ? It looks to me as if the functionality specified for
the former is simpler therefore more likely to work. If you want
to append you use :append. So I still don't see where you'd want
to use :supersede.
> > 2) How would :supersede be implemented ?
>
> This is implementation dependant. But mostly, it does something right.
Yes , I was looking for a description of an example
implementation in order to get a better idea of how :supersede is
supposed to work.
> > I was thinking write to
> > a temporary file and when the stream is closed move the temporary
> > file to its proper location. But how is the implementation
> > supposed to behave if you open a file for writing with :supersede
> > and before closing it you open it again for reading ? Are you
> > supposed to read what you wrote with :supersede or the original
> > content of the file ?
>
> No, it won't work. You will have to read the data before overwritting
> it.
What won't work , the implementation I suggested or opening the
file for reading and getting what was there before the
:supersede ?
> But you could also open the file :direction :io :if-exists :append
> and _modify_ it, reading and writting at will.
>
> > 3) What would a "reasonable" behavior be for an implementation
> > which cannot do the right thing with :supersede ? I was thinking
> > the open should fail so that the programmer knows immediately
> > that the functionality is not available otherwise he may end up
> > irretrievably corrupting the file.
>
> CLHS says:
>
> An implementation is required to recognize all of the open keyword
> options and to do something reasonable in the context of the host
> operating system. For example, if a file system does not support
> distinct file versions and does not distinguish the notions of
> deletion and expunging, :new-version might be treated the same as
> :rename or :supersede, and :rename-and-delete might be treated the
> same as :supersede.
>
> So the behavior would have to be host OS dependant.
Taking as our working assumption that the implementation cannot
handle :supersede as the standard demands then from there on it
doesn't have to be OS dependent. Having open return NIL when
called with :supersede can be achieved in any operating system.
In any case I was asking what behavior you as a programmer would
consider reasonable i.e. a behavior that you don't feel pulls the
rug from under your feet.
> However, there seems that there is some leeway allowing CL
> implementation to "interpret" the host environment, so we can get
> different implementations on the same OS implementing different
> behaviors for platform specific features.
>
> In short, it's a mess.
>
> For example, for your purpose above, :rename-and-delete could be a
> better option.
But the HS says for :rename-and-delete "The existing file is
renamed to some other name, then it is deleted but not
expunged..." This expunged thing suggests to me that it is meant
to work on an operating system which supports file versions.
Otherwise what would the meaning of "deleted but not expunged"
be on Unix or MS Windows ?
> We would need a substandard indicating how pathname, file and stream
> operations should be implemented on POSIX systems. A CDR could be
> written, and implementations could be nudged toward it.
A substandard standard doesn't sound like a good thing :-D
What's a CDR ?
--
The Balkans Intervention achieved all aims rapidly, giving the
European continent its first peace in 4,000 years, dramatically
enhancing America's reputation and world popularity (especially
among Muslims)...
David Brin
Spiros Bousbouras <······@gmail.com> writes:
>> We would need a substandard indicating how pathname, file and stream
>> operations should be implemented on POSIX systems. A CDR could be
>> written, and implementations could be nudged toward it.
>
> A substandard standard doesn't sound like a good thing :-D
Perhaps the term is ill-choosen. I mean a part of the CL standard
that would apply only where appliable, namely on POSIX systems in this
example.
> What's a CDR ?
http://cdr.eurolisp.org/
--
__Pascal Bourguignon__
On Sun, 12 Apr 2009 06:49:34 -0700 (PDT), Spiros Bousbouras
<······@gmail.com> wrote:
>3 questions:
>
>1) What would a realistic scenario be where you'd want to use
>:supersede ?
I think :supercede is meant for versioning file systems like VMS -
where it means "start a new version of the file".
>I was thinking perhaps a situation where you want to modify a file
>in place ...
You'd use :overwrite for that.
>2) How would :supersede be implemented ? I was thinking write to
>a temporary file and when the stream is closed move the temporary
>file to its proper location. But how is the implementation
>supposed to behave if you open a file for writing with :supersede
>and before closing it you open it again for reading ? Are you
>supposed to read what you wrote with :supersede or the original
>content of the file ?
In a versioning system, it would just create a new file with the same
name and (if necessary) tell the OS it is a new version and not a
replacement.
For a non-versioning system with share locking, one way would be to
write lock the whole original file and copy data from the temporary
one. You'd still need coordination with or control of any readers so
they either know the file changed or never assume anything and always
read data fresh before using it.
Without share locking, you'd have to somehow monitor open handles and
replace the original when all handles are closed. But you'd need
cooperation from the OS because Lisp may not have opened all the
handles.
>3) What would a "reasonable" behavior be for an implementation
>which cannot do the right thing with :supersede ? I was thinking
>the open should fail so that the programmer knows immediately
>that the functionality is not available otherwise he may end up
>irretrievably corrupting the file.
The most reasonable thing, I think, is make :supercede a synonym for
":if-exists :rename" and construct a new base name (name less path and
extension if any) by appending a version tag. Of course you have to
check whether the existing name already conforms to your versioning
scheme and either increment the existing tag or add it to the raw file
base name if this is the first time :supercede is used.
George
On 13 Apr, 22:51, George Neuner <········@comcast.net> wrote:
> On Sun, 12 Apr 2009 06:49:34 -0700 (PDT), Spiros Bousbouras
>
> <······@gmail.com> wrote:
> >3 questions:
>
> >1) What would a realistic scenario be where you'd want to use
> >:supersede ?
>
> I think :supercede is meant for versioning file systems like VMS -
> where it means "start a new version of the file".
>
> >I was thinking perhaps a situation where you want to modify a file
> >in place ...
>
> You'd use :overwrite for that.
Would that work ? Say I want to replace every "a" in the file
by "abc". If I open the file with
(open :direction :io :if-exists :overwrite)
and do the straightforward thing i.e. write "abc" whenever I read
"a", wouldn't that overwrite some data before I get a chance to
process it ? Obviously I can read ahead and store what I read in
memory but I was wondering whether :supersede will do the right
thing without extra processing on the part of the programme.
> The most reasonable thing, I think, is make :supercede a synonym for
> ":if-exists :rename" and construct a new base name (name less path and
> extension if any) by appending a version tag. Of course you have to
> check whether the existing name already conforms to your versioning
> scheme and either increment the existing tag or add it to the raw file
> base name if this is the first time :supercede is used.
And if the operating system does not have any provisions for file
versions then the implementation would come up with some
versioning scheme of its own , yes ?
Spiros Bousbouras <······@gmail.com> writes:
> On 13 Apr, 22:51, George Neuner <········@comcast.net> wrote:
>> On Sun, 12 Apr 2009 06:49:34 -0700 (PDT), Spiros Bousbouras
>>
>> <······@gmail.com> wrote:
>> >3 questions:
>>
>> >1) What would a realistic scenario be where you'd want to use
>> >:supersede ?
>>
>> I think :supercede is meant for versioning file systems like VMS -
>> where it means "start a new version of the file".
>>
>> >I was thinking perhaps a situation where you want to modify a file
>> >in place ...
>>
>> You'd use :overwrite for that.
>
> Would that work ? Say I want to replace every "a" in the file
> by "abc". If I open the file with
> (open :direction :io :if-exists :overwrite)
> and do the straightforward thing i.e. write "abc" whenever I read
> "a", wouldn't that overwrite some data before I get a chance to
> process it ? Obviously I can read ahead and store what I read in
> memory but I was wondering whether :supersede will do the right
> thing without extra processing on the part of the programme.
It would not do what you want automatically, you will have to handle
the overwriting yourself.
Note that you didn't specify the file format, it could contain
fixed-size records, and overwriting or inserting/deleting them would
not pose any problem (given the file format). But of course, you or a
library would have to manage it.
>> The most reasonable thing, I think, is make :supercede a synonym for
>> ":if-exists :rename" and construct a new base name (name less path and
>> extension if any) by appending a version tag. Of course you have to
>> check whether the existing name already conforms to your versioning
>> scheme and either increment the existing tag or add it to the raw file
>> base name if this is the first time :supercede is used.
>
> And if the operating system does not have any provisions for file
> versions then the implementation would come up with some
> versioning scheme of its own , yes ?
I wouldn't mind, indeed. I like emacs versionning. But the main
problem is that various implementations don't agree. We could write a
CDR specifying how to implement versions over a POSIX system, and
users could push it to implementers. On the other hand, it may be
argued that automatic (system level) versionning is not a good
feature. I don't know. But as a user, I like versions, and it seems
MacOSX users too, since they re-implement them with TimeMachine.
Perhaps it's still just too soon to standardize anything, and more
experiment is needed? Patch your favorite implementation to behave as
you want, and let's see how good it is.
--
__Pascal Bourguignon__
On Mon, 13 Apr 2009 15:24:00 -0700 (PDT), Spiros Bousbouras
<······@gmail.com> wrote:
>On 13 Apr, 22:51, George Neuner <········@comcast.net> wrote:
>> On Sun, 12 Apr 2009 06:49:34 -0700 (PDT), Spiros Bousbouras
>>
>> <······@gmail.com> wrote:
>> >3 questions:
>>
>> >1) What would a realistic scenario be where you'd want to use
>> >:supersede ?
>>
>> I think :supercede is meant for versioning file systems like VMS -
>> where it means "start a new version of the file".
>>
>> >I was thinking perhaps a situation where you want to modify a file
>> >in place ...
>>
>> You'd use :overwrite for that.
>
>Would that work ? Say I want to replace every "a" in the file
>by "abc". If I open the file with
>(open :direction :io :if-exists :overwrite)
>and do the straightforward thing i.e. write "abc" whenever I read
>"a", wouldn't that overwrite some data before I get a chance to
>process it ? Obviously I can read ahead and store what I read in
>memory but I was wondering whether :supersede will do the right
>thing without extra processing on the part of the programme.
You're maybe expecting a word processor? No. As Pascal B. mentioned
already, if the file is record oriented, you can change individual
records at will (as long as you don't exceed the record length). With
a stream file, though, editing using search/replace with two different
length strings will blow up in you face ... to do it correctly, you
have to approximate reel to reel tape processing.
All :supercede might get you (if you're lucky) is versioning. You
still have to open some existing version of the file for input so
you'll need to know how to specify a particular version of a file -
which is unlikely to be portable.
>> The most reasonable thing, I think, is make :supercede a synonym for
>> ":if-exists :rename" and construct a new base name (name less path and
>> extension if any) by appending a version tag. Of course you have to
>> check whether the existing name already conforms to your versioning
>> scheme and either increment the existing tag or add it to the raw file
>> base name if this is the first time :supercede is used.
>
>And if the operating system does not have any provisions for file
>versions then the implementation would come up with some
>versioning scheme of its own , yes ?
If the OS has native versioning then :supercede should use it.
Otherwise, you are correct that the Lisp implementation would have to
do its own versioning.
George
On Wed, 15 Apr 2009 12:04:30 +0200, Nicolas Neuss
<········@math.uni-karlsruhe.de> wrote:
>George Neuner <········@comcast.net> writes:
>
>> All :supercede might get you (if you're lucky) is versioning. You
> ^^^^^^^^^^
> :supersede
>
>Nicolas
My dictionary lists them as alternate spellings, but I did Google the
etymology and it appears that the 's' spelling is preferred because
the 'c' spelling connotes a slightly different, though not
incompatible, meaning.
Thank you.
George