Locking files for CLASH_DETECTION now supported on MS-Windows

Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

Trunk revision 111877 implements file locking and clash detection for
MS-Windows (see the node "Interlocking" in the Emacs manual for the
details), so that the related features now work on Windows like they
do on Posix platforms.

After you update from bzr, and compile Emacs, I suggest to
byte-compile files.el and userlock.el, and re-dump Emacs, because
otherwise Emacs might "remember" that it didn't have the primitives
for file locking implemented (they are 'ignore'd away in files.el).

If you don't like interlocking for some reason, you can get the old
behavior back by resetting create-lockfiles to nil.

Enjoy.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Paul Eggert]

On 02/25/13 09:43, Eli Zaretskii wrote:
> Trunk revision 111877 implements file locking and clash detection for
> MS-Windows (see the node "Interlocking" in the Emacs manual for the
> details),

The MS-Windows code uses regular files while the GNUish code
uses symbolic links.  Suppose one instance of Emacs is running
on a GNUish host while the other one is running on MS-Windows,
and they are both accessing the same file via NFS.  Won't
that cause problems?  The MS-Windows code will use a regular
file .#FOO while the GNUish code is programmed to ignore locking
for FOO if a regular file .#FOO exists (for compatibility with
non-Emacs programs that use .#FOO for other purposes).

Also, suppose an MS-Windows Emacs uses .#FOO as a lock file while
a non-Emacs program uses .#FOO as a regular file for something else.
Won't there be problems of a different sort?

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Mon, 25 Feb 2013 10:05:42 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> On 02/25/13 09:43, Eli Zaretskii wrote:
> > Trunk revision 111877 implements file locking and clash detection for
> > MS-Windows (see the node "Interlocking" in the Emacs manual for the
> > details),
> 
> The MS-Windows code uses regular files while the GNUish code
> uses symbolic links.  Suppose one instance of Emacs is running
> on a GNUish host while the other one is running on MS-Windows,
> and they are both accessing the same file via NFS.  Won't
> that cause problems?

If MS-Windows creates the lock file first, then the Posix host will
see that it's a regular file and skip it.  If a Posix host creates the
lock file first, things depend on how symlinks are supported via NFS
to Windows.  For example, if Emacs on Windows will see a symlink as an
empty file, then it will skip it.

If someone tries this with NFS and reports results, we will see if
some additional code is needed to avoid problems in this scenario.

> Also, suppose an MS-Windows Emacs uses .#FOO as a lock file while
> a non-Emacs program uses .#FOO as a regular file for something else.
> Won't there be problems of a different sort?

The test in IS_LOCK_FILE might detect that it cannot be a lock file.
If that fails, then Emacs at worst might display a prompt for stealing
the lock with some weirdo user and host name, and at best it will
silently fail to lock the file.  At least this is my reading of
filelock.c and of its callers.

Anyway, file names that start with a period are pretty unusual on
Windows, and a with period followed by a hash sign are extremely rare
outside of the Emacs world.  E.g., I have 73 such files on my disk,
all of them from Emacs sessions.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Paul Eggert]

On 02/25/13 10:41, Eli Zaretskii wrote:
> If MS-Windows creates the lock file first, then the Posix host will
> see that it's a regular file and skip it.

Yes, that's one worrisome scenario -- if an MS-Windows
Emacs locks a file, POSIXish Emacs instances will ignore
the locks and won't be able to set locks themselves.

Two thoughts.  First, how about if the MS-Windows code
tries to use a symbolic link, and if this works then great,
otherwise it falls back on a regular file.  That way,
MS-Windows users who have the Create Symbolic Links right
will interoperate better with POSIXish Emacs.  (Perhaps
the MS-Windows code can try to directly determine whether
it has the Create Symbolic Links right; or perhaps it can
just try 'symlink' and inspect the resulting errno.)

Second, even if an MS-Windows user doesn't have a Create Symbolic
Links right, Emacs should still respect a symlink that happens
to be there, right?  Because the symlink could be placed there
by a POSIXish Emacs.  Does that part work correctly now on
MS-Windows?

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Glenn Morris]

Eli Zaretskii wrote:

> Anyway, file names that start with a period are pretty unusual on
> Windows, and a with period followed by a hash sign are extremely rare
> outside of the Emacs world.  E.g., I have 73 such files on my disk,
> all of them from Emacs sessions.

I have 109 (on a GNU/Linux system), all apparently associated with CVS,
not Emacs.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  emacs-devel@gnu.org
> Date: Mon, 25 Feb 2013 14:39:00 -0500
> 
> Eli Zaretskii wrote:
> 
> > Anyway, file names that start with a period are pretty unusual on
> > Windows, and a with period followed by a hash sign are extremely rare
> > outside of the Emacs world.  E.g., I have 73 such files on my disk,
> > all of them from Emacs sessions.
> 
> I have 109 (on a GNU/Linux system), all apparently associated with CVS,
> not Emacs.

I have those, too.  But I don't think they present any danger, because
they end in telltale .X.Y numeric tails, and are generally larger than
the 1.3KB limit used by filelock.c on Windows to reject non-lockfiles.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Stefan Monnier]

> uses symbolic links.  Suppose one instance of Emacs is running
> on a GNUish host while the other one is running on MS-Windows,
> and they are both accessing the same file via NFS.
> Won't that cause problems?

I wouldn't worry about it, no.  The example would be less hypothetical
if you used SMB rather than NFS, but even so, in the worst case one of
the machines will disable the file-lock-checks, which is no worse than
what we had before (where the Windows machine had the file-lock-checks
disabled).

A much more likely scenario is to see users complain/worry on
gnu.emacs.help about weird files appearing on their systems, then
discovering that come from their new Emacs, and finally asking how to
get rid of "this stupid new feature".


        Stefan

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Paul Eggert]

On 02/25/13 12:35, Stefan Monnier wrote:
> the worst case one of
> the machines will disable the file-lock-checks, which is no worse than
> what we had before (where the Windows machine had the file-lock-checks
> disabled

It depends on one's point of view.  Before, locking worked on
GNU/Linux hosts.  With the change, locking is no longer reliable on
GNU/Linux hosts if an MS-Windows session is active or has crashed.
So GNU/Linux users are worse off than before.  MS-Windows users
are better off of course, but we should strive for a solution
that doesn't also make GNU/Linux users worse off.

> A much more likely scenario is to see users complain/worry on
> gnu.emacs.help about weird files appearing on their systems

Yes, perhaps there should be a way to turn off the use of
regular files for locking on MS-Windows hosts.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Paul Eggert]

One other thought.  Part of the reason GNU/POSIXish hosts use
symbolic links is that readlink, symlink and unlink
are atomic operations.  They let Emacs atomically create
a lock, test whether a lock exists, and remove a lock.  This
doesn't seem to be true of the new MS-Windows implementation,
which uses open+write+close to create a lock and open+read+close
to test whether a lock exists, neither of which is atomic.
Is there some way this can be done atomically on MS-Windows?

Are readlink, symlink, and unlink atomic on MS-Windows?

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

On 2/25/2013 6:03 PM, Paul Eggert wrote:
> One other thought.  Part of the reason GNU/POSIXish hosts use
> symbolic links is that readlink, symlink and unlink
> are atomic operations.  They let Emacs atomically create
> a lock, test whether a lock exists, and remove a lock.  This
> doesn't seem to be true of the new MS-Windows implementation,
> which uses open+write+close to create a lock and open+read+close
> to test whether a lock exists, neither of which is atomic.
> Is there some way this can be done atomically on MS-Windows?
> 
> Are readlink, symlink, and unlink atomic on MS-Windows?

Why would they have to be? File creation is atomic. File deletion is atomic. The
lock is signaled by the presence of the file, right? Why would anything else matter?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Paul Eggert]

On 02/25/2013 06:43 PM, Daniel Colascione wrote:
> File creation is atomic. File deletion is atomic.
> The lock is signaled by the presence of the file, right?

Not in the recently-added MS-Windows implementation, no.
The lock is signaled by a nonempty file (approximately;
the details are a bit more complicated).

There are races where two Emacs instances can both
grab the lock simultaneously.  For example, one Emacs
instance can call create_lock_file with !FORCE and another
with FORCE, and afterwards both Emacs instances
think they own the lock (and the lock file itself
says the !FORCE Emacs owns it).

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 25 Feb 2013 15:35:24 -0500
> 
> A much more likely scenario is to see users complain/worry on
> gnu.emacs.help about weird files appearing on their systems, then
> discovering that come from their new Emacs, and finally asking how to
> get rid of "this stupid new feature".

That part is in NEWS.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Mon, 25 Feb 2013 14:49:46 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Yes, perhaps there should be a way to turn off the use of
> regular files for locking on MS-Windows hosts.

The usual setting of create-lockfiles to nil will do that.  Or did you
mean something else?

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Mon, 25 Feb 2013 10:56:01 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> On 02/25/13 10:41, Eli Zaretskii wrote:
> > If MS-Windows creates the lock file first, then the Posix host will
> > see that it's a regular file and skip it.
> 
> Yes, that's one worrisome scenario -- if an MS-Windows
> Emacs locks a file, POSIXish Emacs instances will ignore
> the locks and won't be able to set locks themselves.

No, I think Emacs running on a Posix host will ignore the Windows lock
file (because it's a regular file) and lock the file itself with a
different file name .#FOO.0.  So on Posix host, the file will be
locked, as it is now.

> Two thoughts.  First, how about if the MS-Windows code
> tries to use a symbolic link, and if this works then great,
> otherwise it falls back on a regular file.  That way,
> MS-Windows users who have the Create Symbolic Links right
> will interoperate better with POSIXish Emacs.

Unfortunately, most Windows users who have this privilege are also
members of the Administrators group, so Windows still pops up the UAC
elevation dialog for them.  It is entirely non-trivial to set Windows
up so that these prompts are avoided.  I don't want to rely on users
to do that.  We will get gobs of complaints that way.

> (Perhaps
> the MS-Windows code can try to directly determine whether
> it has the Create Symbolic Links right; or perhaps it can
> just try 'symlink' and inspect the resulting errno.)

The problem is that the error is returned _after_ prompting.

> Second, even if an MS-Windows user doesn't have a Create Symbolic
> Links right, Emacs should still respect a symlink that happens
> to be there, right?

If NFS reflects symlinks to Windows, then yes.  But I need someone
with good access to NFS from Windows to test such code (I don't have
such access myself).  ANd it will only work on latest versions of
Windows; Windows XP and older doesn't have the support for accessing
symlinks, so 'readlink' implemented on w32.c won't work there (it will
report that a file isn't a symlink).

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Mon, 25 Feb 2013 18:03:13 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> One other thought.  Part of the reason GNU/POSIXish hosts use
> symbolic links is that readlink, symlink and unlink
> are atomic operations.  They let Emacs atomically create
> a lock, test whether a lock exists, and remove a lock.  This
> doesn't seem to be true of the new MS-Windows implementation,
> which uses open+write+close to create a lock and open+read+close
> to test whether a lock exists, neither of which is atomic.
> Is there some way this can be done atomically on MS-Windows?

Why is that important?  On Windows, as long as a file is open, no
other application can write to it or remove the file.  Is that the
"atomic" nature you wanted?

> Are readlink, symlink, and unlink atomic on MS-Windows?

Please define "atomic" for these cases, and I will think about that.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

On 2/25/2013 8:02 PM, Eli Zaretskii wrote:
>> Date: Mon, 25 Feb 2013 18:03:13 -0800
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> CC: emacs-devel@gnu.org
>>
>> One other thought.  Part of the reason GNU/POSIXish hosts use
>> symbolic links is that readlink, symlink and unlink
>> are atomic operations.  They let Emacs atomically create
>> a lock, test whether a lock exists, and remove a lock.  This
>> doesn't seem to be true of the new MS-Windows implementation,
>> which uses open+write+close to create a lock and open+read+close
>> to test whether a lock exists, neither of which is atomic.
>> Is there some way this can be done atomically on MS-Windows?
> 
> Why is that important?  On Windows, as long as a file is open, no
> other application can write to it or remove the file.  Is that the
> "atomic" nature you wanted?
> 
>> Are readlink, symlink, and unlink atomic on MS-Windows?
> 
> Please define "atomic" for these cases, and I will think about that.

If we really need atomicity under Windows, Transactional NTFS will work for more
users than symbolic links will.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Paul Eggert]

>> Yes, that's one worrisome scenario -- if an MS-Windows
>> Emacs locks a file, POSIXish Emacs instances will ignore
>> the locks and won't be able to set locks themselves.
> 
> No, I think Emacs running on a Posix host will ignore the Windows lock
> file (because it's a regular file) and lock the file itself with a
> different file name .#FOO.0.  So on Posix host, the file will be
> locked, as it is now.

Sorry, I should have been clearer.  My comment was assuming the
patch in Bug#13807, which simplifies the analysis.  In the
current Emacs the races are different and the analysis trickier,
but the bugs are still there.

For example, assuming current trunk (bzr 111882), suppose MS-Windows
host A creates a regular file '.#FOO', and GNU/POSIX host B creates a
symbolic link '.#FOO.0'.  They both think they own the lock for FOO,
which is bad but no worse than Emacs 24.3.  But suppose that A then
releases its lock and GNU/POSIX host C then locks the file with a
symbolic link '.#FOO'.  At this point B and C both think they own the
lock, which is a regression from Emacs 24.3.  The bug exists because
of A's intervention and because of the new MS-Windows behavior.

There are other races like that.  The Bug#13807 patch fixes the races
for GNU/POSIX hosts.  But there are still races if MS-Windows hosts are
involved.

>> (Perhaps
>> the MS-Windows code can try to directly determine whether
>> it has the Create Symbolic Links right; or perhaps it can
>> just try 'symlink' and inspect the resulting errno.)
> 
> The problem is that the error is returned _after_ prompting.

OK, then how about just checking whether the privilege is present?
That shouldn't cause a prompt.  For example, Emacs might invoke
OpenProcessToken and GetTokenInformation to see whether the current
process has SeCreateSymbolicLinkPrivilege.  Or it could do something
simpler and slower, e.g., run the equivalent of system ("WHOAMI /PRIV
| grep 'SeCreateSymbolicLinkPrivilege.*Enabled'") (pardon my lack of
MS-Windows coding skills, but I hope you get the idea).

>> Second, even if an MS-Windows user doesn't have a Create Symbolic
>> Links right, Emacs should still respect a symlink that happens
>> to be there, right?
> 
> If NFS reflects symlinks to Windows, then yes.

Apparently it does, but the symlinks appear to be regular files (!).
See <http://technet.microsoft.com/en-us/library/hh310285%28v=ws.10%29.aspx>
and <http://msdn.microsoft.com/en-us/library/ee380665.aspx>.

This whole area is messy, alas.

How about this idea?  It should be an improvement:
the MS-Windows port could use symbolic links a la the patch in Bug#13807
if SeCreateSymbolicLinkPrivilege is present, and fall back
on the 24.3 behavior if SeCreateSymbolicLinkPrivilege is absent.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Paul Eggert]

On 02/25/2013 08:02 PM, Eli Zaretskii wrote:> 

> On Windows, as long as a file is open, no
> other application can write to it or remove the file.

Is this true even if the file is network-mounted via NFS or CIFS?
I'd be surprised if it were true for NFS, as that's not the
standard NFS semantics.

> Is that the "atomic" nature you wanted?

Partly, but it's not sufficient.  I assume that one process can
read a file while another is writing to it, which means that
the readlink and symlink replacements may not be atomic.

>> Are readlink, symlink, and unlink atomic on MS-Windows?
> 
> Please define "atomic" for these cases, and I will think about that.

I mean that there's an abstract machine where a symbolic link is
either present or absent, and where readlink ("a", ...),
symlink ("b", "a"), and unlink ("a") either succeed or fail and affect
the abstract machine in an all-or-nothing way.  For example, if
the symbolic link doesn't exist and one process runs
symlink ("very long link contents", "a")
at the same time another process runs readlink ("a", buf, sizeof buf)
where buf is large enough, then the readlink should either fail
(no symlink present yet) or should succeed with
the buffer containing all the link contents; it shouldn't succeed with
partial contents such as "very long link co".

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Richard Stallman]

    It depends on one's point of view.  Before, locking worked on
    GNU/Linux hosts.  With the change, locking is no longer reliable on
    GNU/Linux hosts if an MS-Windows session is active or has crashed.
    So GNU/Linux users are worse off than before.  MS-Windows users
    are better off of course, but we should strive for a solution
    that doesn't also make GNU/Linux users worse off.

We must not make things worse for GNU/Linux users
to cater to Windows users.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Mon, 25 Feb 2013 20:28:35 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel@gnu.org
> 
> If we really need atomicity under Windows, Transactional NTFS will work for more
> users than symbolic links will.

No, I don't think we want Transactional NTFS (or TxF for short), for
several good reasons.

First, Microsoft itself is considering deprecating TxF, and strongly
recommends that developers use alternatives.  A quotation from
http://msdn.microsoft.com/en-us/library/windows/desktop/hh802690%28v=vs.85%29.aspx,
which speaks for itself:

  Abstract

  Microsoft strongly recommends developers investigate utilizing the
  discussed alternatives (or in some cases, investigate other
  alternatives) rather than adopting an API platform which may not be
  available in future versions of Windows.

  Introduction

  TxF was introduced with Windows Vista as a means to introduce atomic
  file transactions to Windows. It allows for Windows developers to have
  transactional atomicity for file operations in transactions with a
  single file, in transactions involving multiple files, and in
  transactions spanning multiple sources - such as the Registry (through
  TxR), and databases (such as SQL). While TxF is a powerful set of
  APIs, there has been extremely limited developer interest in this API
  platform since Windows Vista primarily due to its complexity and
  various nuances which developers need to consider as part of
  application development. As a result, Microsoft is considering
  deprecating TxF APIs in a future version of Windows to focus
  development and maintenance efforts on other features and APIs which
  have more value to a larger majority of customers.

Second, TxF is not really meant for atomicity in the sense that we are
looking for in this thread.  It is meant to preserve consistency in a
scenario of a catastrophic failure, such as an application or system
crash.  E.g., this page:

  http://msdn.microsoft.com/en-us/library/windows/desktop/aa365738%28v=vs.85%29.aspx

clearly says near its end that several writers to the same file --
exactly what bothers us here -- is not a supported scenario by TxF.
Read this page:

  http://msdn.microsoft.com/en-us/library/windows/desktop/aa964931%28v=vs.85%29.aspx

to see how TxF is designed for scenarios that are very different from
what we are considering here.  E.g.:

  Reparse Points

  Changes to reparse points are transacted, which means that if a new
  reparse point gets assigned to a file in a transaction, it is not
  visible to the other transactions. Similarly, changes or removal of an
  existing reparse point are not visible until commit.

(NTFS symlinks are a kind of reparse point.)  This is exactly the
opposite of what we would like to happen.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Mon, 25 Feb 2013 21:46:21 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> >> (Perhaps
> >> the MS-Windows code can try to directly determine whether
> >> it has the Create Symbolic Links right; or perhaps it can
> >> just try 'symlink' and inspect the resulting errno.)
> > 
> > The problem is that the error is returned _after_ prompting.
> 
> OK, then how about just checking whether the privilege is present?

That doesn't help, because according to my testing, the privilege is
normally not originally present in the process token, even if the user
can acquire that privilege.  (The w32 'symlink' implementation tries
to acquire the privilege if the attempt to create a symlink without
doing that fails.)

> >> Second, even if an MS-Windows user doesn't have a Create Symbolic
> >> Links right, Emacs should still respect a symlink that happens
> >> to be there, right?
> > 
> > If NFS reflects symlinks to Windows, then yes.
> 
> Apparently it does, but the symlinks appear to be regular files (!).
> See <http://technet.microsoft.com/en-us/library/hh310285%28v=ws.10%29.aspx>
> and <http://msdn.microsoft.com/en-us/library/ee380665.aspx>.

Yes, my testing on one particular NFS-mounted volume confirms this:
Emacs on Windows sees Unix-side symlinks as regular files.  Reading
them yields some magic signature, followed by something that looks
like the target of the symlink encoded in UTF-16.  Is this the
universally accepted way of writing a symlink?

> How about this idea?  It should be an improvement:
> the MS-Windows port could use symbolic links a la the patch in Bug#13807
> if SeCreateSymbolicLinkPrivilege is present, and fall back
> on the 24.3 behavior if SeCreateSymbolicLinkPrivilege is absent.

See above.  It would be easy to allow the w32 Emacs try 'readlink'
first, and only fall back on reading the file if that fails, though.
That way, if some remote filesystem does support symlinks, at least
part of the problems with Windows/Unix interoperability will be
avoided.

In any case, since Windows support for symlinks across NFS mounts is
very incomplete, to say the least, using symlinks does not seem to be
a reliable way of avoiding potential problems to Emacs running on
Posix hosts by Emacs running on Windows.  So I suggest we look into
alternative solutions for these problems.  I will write about that in
a separate message.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Mon, 25 Feb 2013 10:56:01 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> On 02/25/13 10:41, Eli Zaretskii wrote:
> > If MS-Windows creates the lock file first, then the Posix host will
> > see that it's a regular file and skip it.
> 
> Yes, that's one worrisome scenario -- if an MS-Windows
> Emacs locks a file, POSIXish Emacs instances will ignore
> the locks and won't be able to set locks themselves.

We could avoid this by teaching Emacs on Posix systems to read lock
files created by Emacs running on Windows.  All this takes is losing
the #ifdef's in read_lock_data, the code there is general and not
Windows specific in any way.  We could also strengthen this by
considering a file locked just because the .#FOO lockfile exists, even
if we cannot read that file or decipher its contents -- this would
avoid losing due to some race conditions.  We could, e.g., tell the
Posix user that the file "appears to be locked by an unknown remote
user" or some such, if we cannot parse the lock information.

Another possibility would be to make Emacs running on Windows avoid
locking files on remote filesystems.  AFAIU, failure to lock a file is
silently ignored.  This method has a disadvantage of including in the
ban networked volumes run by Windows servers, where locking could be
enabled, but I think there are APIs which will allow to discern
between the two.  And even if we cannot distinguish between them, the
situation won't be worse than in previous versions of Emacs on
Windows.

> For example, assuming current trunk (bzr 111882), suppose MS-Windows
> host A creates a regular file '.#FOO', and GNU/POSIX host B creates a
> symbolic link '.#FOO.0'.  They both think they own the lock for FOO,
> which is bad but no worse than Emacs 24.3.  But suppose that A then
> releases its lock and GNU/POSIX host C then locks the file with a
> symbolic link '.#FOO'.  At this point B and C both think they own the
> lock, which is a regression from Emacs 24.3.  The bug exists because
> of A's intervention and because of the new MS-Windows behavior.

This scenario would be solved by any of the two possible solutions
mentioned above, I think.

Are there any other scenarios we should consider?

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Mon, 25 Feb 2013 22:05:15 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> On 02/25/2013 08:02 PM, Eli Zaretskii wrote:> 
> 
> > On Windows, as long as a file is open, no
> > other application can write to it or remove the file.
> 
> Is this true even if the file is network-mounted via NFS or CIFS?

It is true if the other process runs on the same machine, I think.

> > Is that the "atomic" nature you wanted?
> 
> Partly, but it's not sufficient.  I assume that one process can
> read a file while another is writing to it

That depends on how each process opens the file, exactly.  It can be
arranged that no other process can open the same file, although I
think the default open flags don't do that.

> which means that the readlink and symlink replacements may not be
> atomic.

Don't know about this part, I will have to run some tests.  Since
changes in file metadata _are_ atomic on NTFS (see below) the answer
to that appears to depend on whether NTFS symlinks are implemented as
metadata or as data.  I don't know which is the case, for now.

> I mean that there's an abstract machine where a symbolic link is
> either present or absent, and where readlink ("a", ...),
> symlink ("b", "a"), and unlink ("a") either succeed or fail and affect
> the abstract machine in an all-or-nothing way.  For example, if
> the symbolic link doesn't exist and one process runs
> symlink ("very long link contents", "a")
> at the same time another process runs readlink ("a", buf, sizeof buf)
> where buf is large enough, then the readlink should either fail
> (no symlink present yet) or should succeed with
> the buffer containing all the link contents; it shouldn't succeed with
> partial contents such as "very long link co".

It's hard to give a definitive answer to that, as documentation is
scarce and most of it unofficial.  My conclusion from reading a lot of
technical stuff is that changes to file metadata are atomic on NTFS,
and single-sector read/write operations are also atomic.  That means
'unlink' (file deletion) is atomic, as it affects metadata only, and
so is 'rename' and 'chmod'.  Reads and writes are probably atomic as
long as they are short.  I found nothing about the APIs used by
'symlink' and 'readlink', so all I can say that I hope they are
atomic, because the amount of data they read/write is small.  I will
try to test this when I have time.

In any case, all this atomicity stuff is important for locking files
on Windows volumes, it won't help us with the more important problems
related to Posix and Windows hosts editing the same file on a
networked filesystem.  We could (and should) probably make the Windows
implementation of locking more robust in the face of potential races
between several instances of Emacs running on Windows, but that is
secondary to avoiding problems for Emacs running on Posix hosts.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Tue, 26 Feb 2013 20:41:34 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Date: Mon, 25 Feb 2013 22:05:15 -0800
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > CC: emacs-devel@gnu.org
> > 
> > On 02/25/2013 08:02 PM, Eli Zaretskii wrote:> 
> > 
> > > On Windows, as long as a file is open, no
> > > other application can write to it or remove the file.
> > 
> > Is this true even if the file is network-mounted via NFS or CIFS?
> 
> It is true if the other process runs on the same machine, I think.

Actually, I see that this is generally true only for deleting and
renaming a file that is open.  If we want to prevent other writers or
readers, we need to open the file with special flags.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Paul Eggert]

On 02/26/13 10:17, Eli Zaretskii wrote:

> We could avoid this by teaching Emacs on Posix systems to
> read lock files created by Emacs running on Windows.

I don't see how to do this without introducing race
conditions that are not present in the current GNU/Linux
implementation.  There are atomic system calls to create,
test, and remove symbolic links, but the corresponding
syscalls do not exist directly for regular files, and would
need to be implemented carefully.  Doing this portably and
reliably, in a way where the MS-Windows and non-MS-Windows
approaches work together, will take some work.

> We could also strengthen this by considering a file locked
> just because the .#FOO lockfile exists

Other applications (i.e., not GNU Emacs) create regular
files .#FOO, and Emacs shouldn't be impeded from editing FOO
merely because some other application has created .#FOO.  (We
could try to parse .#FOO but this leads to races.)

> Another possibility would be to make Emacs running on Windows avoid
> locking files on remote filesystems.

Won't there still be a problem if the local file system is
exported?  MS-Windows Emacs will use Windows-style locking
on the local file system, and this might confuse
non-MS-Windows Emacs accessing the same file.

> Are there any other scenarios we should consider?

To move things forward, I propose that the MS-Windows
implementation use a different lock file name .#-FOO.  (The
'-' is because '-' is not commonly used as the first
character of a file name.)  MS-Windows and non-MS-Windows
instances will ignore each others' locks.  This avoids the
race conditions mentioned earlier, and it will mean
that MS-Windows instances won't mess up non-MS-Windows
instances.  It has the downside that MS-Windows and
non-MS-Windows instances will ignore each others' locks,
but that's better than where we are now.

If we can come up with a reliable scheme whereby MS-Windows
and non-MS-Windows Emacs versions can both use the same lock
file name, we can unify the two approaches, but for now
using different names is a simple fix that should make
GNU/Linux Emacs locking reliable again.

A patch to implement this proposal is in
<http://bugs.gnu.org/13807#14>.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Tue, 26 Feb 2013 14:34:29 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> On 02/26/13 10:17, Eli Zaretskii wrote:
> 
> > We could avoid this by teaching Emacs on Posix systems to
> > read lock files created by Emacs running on Windows.
> 
> I don't see how to do this without introducing race
> conditions that are not present in the current GNU/Linux
> implementation.

Can you describe those race conditions?  I'm talking only about
_reading_ those files, not about creating them.  If two instances of
Emacs read the same lock file (created by an instance running on
Windows), they will both see that the file exists and will both read
the same contents.

> > We could also strengthen this by considering a file locked
> > just because the .#FOO lockfile exists
> 
> Other applications (i.e., not GNU Emacs) create regular
> files .#FOO, and Emacs shouldn't be impeded from editing FOO
> merely because some other application has created .#FOO.

??? The changes you are promoting cause Emacs to take control of any
file whose name matches the pattern .#FILE, where FILE is the file we
edit.  How is that different from what I propose?

> (We could try to parse .#FOO but this leads to races.)

None of the races that doesn't already exist.

> > Another possibility would be to make Emacs running on Windows avoid
> > locking files on remote filesystems.
> 
> Won't there still be a problem if the local file system is
> exported?

No, because such exported systems will not support symlinks from Posix
hosts.

> To move things forward, I propose that the MS-Windows
> implementation use a different lock file name .#-FOO.

I don't mind, but I thought we could do better.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Paul Eggert]

>>> We could avoid this by teaching Emacs on Posix systems to
>>> read lock files created by Emacs running on Windows.
>>
>> I don't see how to do this without introducing race
>> conditions that are not present in the current GNU/Linux
>> implementation.
> 
> Can you describe those race conditions?  I'm talking only about
> _reading_ those files, not about creating them.

Sorry, I must have misunderstood you.
There are no races if access is read-only.  We need at least one
writer (i.e., one process that's either creating or removing a
lock file) in order to have a race.  I thought the original
scenario involved an MS-Windows Emacs locking a file at about
the same time that a GNU/Linux Emacs locks the same file.

> The changes you are promoting cause Emacs to take control of any
> file whose name matches the pattern .#FILE, where FILE is the file we edit.

No, under that proposal, if there's an existing non-symlink
.#FILE then Emacs doesn't lock FILE; it simply bypasses
locking for FILE for entirely and it doesn't modify .#FILE
at all.  This should be relatively rare.

It is true that if there's an existing symlink .#FILE then Emacs
will take control of it when it edits FILE, but that should be OK.

>>> Another possibility would be to make Emacs running on Windows avoid
>>> locking files on remote filesystems.
>>
>> Won't there still be a problem if the local file system is
>> exported?
> 
> No, because such exported systems will not support symlinks from Posix
> hosts.

I thought that such symlinks were supported: the lock would appear to be a regular file
on the MS-Windows server side, but as exported over the network it would
appear to be a symbolic link on the GNU/Linux client side, as that's
how the Microsoft server software works.  If the MS-Windows
and GNU/Linux locks have the same name, they'd collide
I presume.

>> To move things forward, I propose that the MS-Windows
>> implementation use a different lock file name .#-FOO.
> 
> I don't mind, but I thought we could do better.

It's possible that we can do better, but I'm afraid that'll
take some work.  In the meantime I plan to go ahead with that
proposal as it's better than what we have now.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Tue, 26 Feb 2013 14:34:29 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> To move things forward, I propose that the MS-Windows
> implementation use a different lock file name .#-FOO.  (The
> '-' is because '-' is not commonly used as the first
> character of a file name.)  MS-Windows and non-MS-Windows
> instances will ignore each others' locks.  This avoids the
> race conditions mentioned earlier, and it will mean
> that MS-Windows instances won't mess up non-MS-Windows
> instances.  It has the downside that MS-Windows and
> non-MS-Windows instances will ignore each others' locks,
> but that's better than where we are now.
> 
> If we can come up with a reliable scheme whereby MS-Windows
> and non-MS-Windows Emacs versions can both use the same lock
> file name, we can unify the two approaches, but for now
> using different names is a simple fix that should make
> GNU/Linux Emacs locking reliable again.

I just installed a changeset which makes the open/write/close sequence
in create_lock_file atomic wrt other processes that access the same
lock file.  My testing indicates that the lock file cannot be accessed
(read, written, renamed, or deleted) neither from Windows nor from
GNU/Linux, until the Emacs process which is writing the lock file
finishes and closes the handle.

As far as I understand the logic of lock_file, it will silently
refrain from creating a lock file if trying to do so or trying to read
the lock data fails with any error but EEXIST.  After the changes I
made, Emacs on GNU/Linux returns EACCES (theoretically, it could also
be EINVAL, but I didn't see that) if it tries to lock the same file
while Emacs on Windows is in the process of writing its lock file.  In
the reverse scenario, i.e. if a Posix host locks the file and Emacs on
Windows tries to do that later, it fails to read the file's contents
(current_lock_owner returns -1).  In both cases, the Emacs instance
that is the second one to lock the file does not lock it.  I think
this is better than letting both instances lock the same file and
think they each have an exclusive lock.

Re: Locking files for CLASH_DETECTION now supported on MS-Windows

[Eli Zaretskii]

> Date: Tue, 26 Feb 2013 23:45:38 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> >> To move things forward, I propose that the MS-Windows
> >> implementation use a different lock file name .#-FOO.
> > 
> > I don't mind, but I thought we could do better.
> 
> It's possible that we can do better, but I'm afraid that'll
> take some work.

Please see if my latest changes achieve that goal.