bug#71074: 29.3; When doing a backup, the file is missing during interactive questions

bug#71074: 29.3; When doing a backup, the file is missing during interactive questions

[Vincent Lefevre]

When doing a backup, GNU Emacs renames the file (according to strace).
Under some conditions, the file can be lost.

I noticed that due to a bug in the combination between Emacs and gpg,
where the "save" operation cannot be completed.

To reproduce with "emacs -Q", do for a file that is not under version
control (otherwise --eval="(setq vc-make-backup-files t)" is needed):

1. emacs -Q file.gpg
2. Modify the file.
3. Save with C-x C-s

At this point, gpg gets interactive, with a question

  Untrusted key [...].  Use anyway? (y or n)

and one can see that the file "file.gpg" has been renamed to
"file.gpg~".

And it may happen that the save operation will never be completed,
e.g. because of a bug like

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1071372

(where gpg never terminates) or because Emacs may be killed before
the user could complete the save operation.

In short, the file may have silently disappeared. There is a backup,
but the backups may be cleaned up before the user realizes the issue.

Note: I put the backups in a specific directory with

(defun find-backup-file-name (fn)
       (list (concat "~/.poub/" (file-name-nondirectory fn) "~")))

but I look at it only when I know I've done something wrong or if
I get an error message about the save operation. Having a missing
file during interactive questions is bad.

Note: I'm using Debian's package emacs-gtk 1:29.3+1-3.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#71074: 29.3; When doing a backup, the file is missing during interactive questions

[Eli Zaretskii]

> Date: Mon, 20 May 2024 03:07:48 +0200
> From: Vincent Lefevre <vincent@vinc17.net>
> 
> When doing a backup, GNU Emacs renames the file (according to strace).
> Under some conditions, the file can be lost.
> 
> I noticed that due to a bug in the combination between Emacs and gpg,
> where the "save" operation cannot be completed.
> 
> To reproduce with "emacs -Q", do for a file that is not under version
> control (otherwise --eval="(setq vc-make-backup-files t)" is needed):
> 
> 1. emacs -Q file.gpg
> 2. Modify the file.
> 3. Save with C-x C-s
> 
> At this point, gpg gets interactive, with a question
> 
>   Untrusted key [...].  Use anyway? (y or n)
> 
> and one can see that the file "file.gpg" has been renamed to
> "file.gpg~".
> 
> And it may happen that the save operation will never be completed,
> e.g. because of a bug like
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1071372
> 
> (where gpg never terminates) or because Emacs may be killed before
> the user could complete the save operation.

Does it help to customize the variable backup-by-copying to a non-nil
value?

bug#71074: 29.3; When doing a backup, the file is missing during interactive questions

[Vincent Lefevre]

On 2024-05-20 14:30:56 +0300, Eli Zaretskii wrote:
> Does it help to customize the variable backup-by-copying to a non-nil
> value?

Yes, but note that a useless backup is done because the file will
not necessarily be saved (if the save operation is canceled via
an interactive question or due to Debian bug 1071372).

Whether renaming or copying, the backup should be done just before
actually saving.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#71074: 29.3; When doing a backup, the file is missing during interactive questions

[Eli Zaretskii]

> Date: Tue, 21 May 2024 10:44:16 +0200
> From: Vincent Lefevre <vincent@vinc17.net>
> Cc: 71074@debbugs.gnu.org
> 
> On 2024-05-20 14:30:56 +0300, Eli Zaretskii wrote:
> > Does it help to customize the variable backup-by-copying to a non-nil
> > value?
> 
> Yes, but note that a useless backup is done because the file will
> not necessarily be saved (if the save operation is canceled via
> an interactive question or due to Debian bug 1071372).

Sure, but if the danger is to lose the file due to some calamity, a
useless backup is a very minor disadvantage, don't you agree?

> Whether renaming or copying, the backup should be done just before
> actually saving.

We already do that.  However, backup+save is not an atomic operation,
and as long as it is not atomic, there's always a window of
opportunity for some catastrophe, like a system crash or Emacs being
killed, to happen in-between.

Moreover, Emacs being Emacs, with its high degree of customization,
some feature can customize either backup or save (or both) in a way
that makes the above-mentioned window very wide.  That's what happened
in the case you reported: epa-file overrides write-contents, the Emacs
primitive that actually writes the buffer to a file, with its own
version, and that's why you get that prompt about untrusted key.  We
cannot avoid such situations, because if we disallow customizations
like that, Emacs will no longer be Emacs.  In fact, you yourself use a
similar feature, when you define find-backup-file-name to force all
the backup files to go to a specific directory.  Such a function could
in theory do anything it wants, including prompting you and whatnot,
thus prolonging the window between the backup and the save.

So I don't see how we could significantly change this situation
without losing very useful features, and therefore I think
backup-by-copying is the best solution we can provide in such cases.

bug#71074: 29.3; When doing a backup, the file is missing during interactive questions

[Eli Zaretskii]

> Cc: 71074@debbugs.gnu.org
> Date: Tue, 21 May 2024 15:12:05 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> So I don't see how we could significantly change this situation
> without losing very useful features, and therefore I think
> backup-by-copying is the best solution we can provide in such cases.

Actually, an alternative is to customize file-precious-flag to a
non-nil value.

bug#71074: 29.3; When doing a backup, the file is missing during interactive questions

[Vincent Lefevre]

On 2024-05-21 15:12:05 +0300, Eli Zaretskii wrote:
> > Date: Tue, 21 May 2024 10:44:16 +0200
> > From: Vincent Lefevre <vincent@vinc17.net>
> > Cc: 71074@debbugs.gnu.org
> > 
> > On 2024-05-20 14:30:56 +0300, Eli Zaretskii wrote:
> > > Does it help to customize the variable backup-by-copying to a non-nil
> > > value?
> > 
> > Yes, but note that a useless backup is done because the file will
> > not necessarily be saved (if the save operation is canceled via
> > an interactive question or due to Debian bug 1071372).
> 
> Sure, but if the danger is to lose the file due to some calamity, a
> useless backup is a very minor disadvantage, don't you agree?

Yes, but this could be a future improvement.

> > Whether renaming or copying, the backup should be done just before
> > actually saving.
> 
> We already do that.

By "just before", I mean that there should not be any system call
between a backup and the write of the file.

BTW, there is actually an issue with the backup system: the backup
is done only before the *first* save-buffer, while an issue might
appear only for a subsequent save-buffer.

> However, backup+save is not an atomic operation, and as long as it
> is not atomic, there's always a window of opportunity for some
> catastrophe, like a system crash or Emacs being killed, to happen
> in-between.

Yes, but if the window is restricted to the minimum, the probability
of a system crash / killed Emacs is significantly reduced. I suppose
that file-precious-flag as you suggested later may be the ultimate
solution.

> Moreover, Emacs being Emacs, with its high degree of customization,
> some feature can customize either backup or save (or both) in a way
> that makes the above-mentioned window very wide.  That's what happened
> in the case you reported: epa-file overrides write-contents, the Emacs
> primitive that actually writes the buffer to a file, with its own
> version, and that's why you get that prompt about untrusted key.

So, does this mean that file-precious-flag doesn't work here?

But perhaps that's a bug in epa-file, which should ensure that by
default, the pathname isn't removed (due to the backup by rename)
until the new file can be written. This means that it currently
does the backup too early.

> We cannot avoid such situations, because if we disallow customizations
> like that, Emacs will no longer be Emacs.  In fact, you yourself use a
> similar feature, when you define find-backup-file-name to force all
> the backup files to go to a specific directory.  Such a function could
> in theory do anything it wants, including prompting you and whatnot,
> thus prolonging the window between the backup and the save.

But customizations should be implemented properly to avoid issues.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#71074: 29.3; When doing a backup, the file is missing during interactive questions

[Eli Zaretskii]

> Date: Wed, 22 May 2024 13:08:18 +0200
> From: Vincent Lefevre <vincent@vinc17.net>
> Cc: 71074@debbugs.gnu.org
> 
> > > Whether renaming or copying, the backup should be done just before
> > > actually saving.
> > 
> > We already do that.
> 
> By "just before", I mean that there should not be any system call
> between a backup and the write of the file.

I don't see how this could be done in Emacs, sorry.  But if you or
someone has practical ideas, let's hear them.

> BTW, there is actually an issue with the backup system: the backup
> is done only before the *first* save-buffer, while an issue might
> appear only for a subsequent save-buffer.

This is how backups in Emacs work.  Catastrophes do happen, but at
least Emacs attempts to auto-save everything when it is about to
crash.  It doesn't always work, depending on the reason for the crash.

> > Moreover, Emacs being Emacs, with its high degree of customization,
> > some feature can customize either backup or save (or both) in a way
> > that makes the above-mentioned window very wide.  That's what happened
> > in the case you reported: epa-file overrides write-contents, the Emacs
> > primitive that actually writes the buffer to a file, with its own
> > version, and that's why you get that prompt about untrusted key.
> 
> So, does this mean that file-precious-flag doesn't work here?

AFAICT, it does work.  Its purpose is to avoid overwriting the
original file until the new stuff was successfully written to disk.

> > We cannot avoid such situations, because if we disallow customizations
> > like that, Emacs will no longer be Emacs.  In fact, you yourself use a
> > similar feature, when you define find-backup-file-name to force all
> > the backup files to go to a specific directory.  Such a function could
> > in theory do anything it wants, including prompting you and whatnot,
> > thus prolonging the window between the backup and the save.
> 
> But customizations should be implemented properly to avoid issues.

I don't see how.  E.g., if we let users and packages customize how
files are backed up, it's anyone's guess what will happen inside the
backup-buffer call and how much time will that take.  What would be a
"proper implementation" that still allows such customizations?  But
again, practical ideas are welcome.

bug#71074: 29.3; When doing a backup, the file is missing during interactive questions

[Vincent Lefevre]

On 2024-05-22 16:25:24 +0300, Eli Zaretskii wrote:
> > Date: Wed, 22 May 2024 13:08:18 +0200
> > From: Vincent Lefevre <vincent@vinc17.net>
> > Cc: 71074@debbugs.gnu.org
> > 
> > > > Whether renaming or copying, the backup should be done just before
> > > > actually saving.
> > > 
> > > We already do that.
> > 
> > By "just before", I mean that there should not be any system call
> > between a backup and the write of the file.
> 
> I don't see how this could be done in Emacs, sorry.  But if you or
> someone has practical ideas, let's hear them.
> 
> > BTW, there is actually an issue with the backup system: the backup
> > is done only before the *first* save-buffer, while an issue might
> > appear only for a subsequent save-buffer.
> 
> This is how backups in Emacs work.  Catastrophes do happen, but at
> least Emacs attempts to auto-save everything when it is about to
> crash.  It doesn't always work, depending on the reason for the crash.

OK, this is actually what I expected to get with the gpg bug. Since
the file.gpg file was not restored from the "save-buffer" failure,
I expected to get a "#file.gpg#" file for recovery, but there was
none. Alternatively, Emacs could have restored the file from the
backup.

I don't what happened, but I don't see the logic behind Emacs.
For instance, when editing a normal file with the Emacs GUI and
doing Ctrl-C in the terminal before saving, a recovery file is
created. But nothing similar is done with .gpg file (of course,
it is not possible to have the unsaved contents because encryption
could not be done, but I would expect at least the saved-encrypted
version back in this case, instead of neither the file nor a
recovery file).

> > > Moreover, Emacs being Emacs, with its high degree of customization,
> > > some feature can customize either backup or save (or both) in a way
> > > that makes the above-mentioned window very wide.  That's what happened
> > > in the case you reported: epa-file overrides write-contents, the Emacs
> > > primitive that actually writes the buffer to a file, with its own
> > > version, and that's why you get that prompt about untrusted key.
> > 
> > So, does this mean that file-precious-flag doesn't work here?
> 
> AFAICT, it does work.  Its purpose is to avoid overwriting the
> original file until the new stuff was successfully written to disk.

BTW, it leaves empty files, for instance, in case of gpg error:

-rw------- 1 vlefevre vlefevre 0 2024-05-23 15:55:36 tmpSHlCS3
-rw------- 1 vlefevre vlefevre 0 2024-05-23 15:57:00 tmpf5O5RM

> > > We cannot avoid such situations, because if we disallow customizations
> > > like that, Emacs will no longer be Emacs.  In fact, you yourself use a
> > > similar feature, when you define find-backup-file-name to force all
> > > the backup files to go to a specific directory.  Such a function could
> > > in theory do anything it wants, including prompting you and whatnot,
> > > thus prolonging the window between the backup and the save.
> > 
> > But customizations should be implemented properly to avoid issues.
> 
> I don't see how.  E.g., if we let users and packages customize how
> files are backed up, it's anyone's guess what will happen inside the
> backup-buffer call and how much time will that take.  What would be a
> "proper implementation" that still allows such customizations?  But
> again, practical ideas are welcome.

I think that this would be more or less equivalent to the
file-precious-flag feature. I'm wondering why it is not enabled
by default. It seems that the only "issue" is that this breaks
the hard links. But hard links seem very rare in practice. If
this is really an issue, the file-precious-flag behavior could
be conditional to whether the file has hard links.

And about "As a side effect, backups are necessarily made by copying.",
another possibility could be proposed: a backup by a hardlink (if
supported). Since the hardlink will normally be broken, this will
normally give no hardlinks at the end, as expected.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)