bug#54126: 29.0.50; C-x x g doesn't always correctly revert SSHFS files

bug#54126: 29.0.50; C-x x g doesn't always correctly revert SSHFS files

[Philipp Stephani]

At least on my system the following happens often, but not always:

Create some file whose contents don't really matter.  In my case:

$ cat /tmp/a.c
int main(void) {
  return 0;
}

Visit the file over SSHFS:

$ emacs -Q /sshfs:localhost:/tmp/a.c

Now, outside of Emacs, append something to the file:

$ echo aaaaa >> /tmp/a.c

Immediately after that, back in Emacs, hit C-x x g.  The new content
isn't there.  Only after reverting the buffer a second time it appears.
First I thought this was a timing/cache coherency issue, but even
waiting for 10 seconds doesn't fix it in most cases.  Somewhat
surprisingly, switching to a different buffer in between appears to make
the problem go away (in some cases at least).


In GNU Emacs 29.0.50 (build 59, x86_64-pc-linux-gnu, GTK+ Version 3.24.31, cairo version 1.16.0)
 of 2022-02-23
Repository revision: 85ad8616007e286c237bb2906d1928bb551462e7
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Debian GNU/Linux rodete

Configured using:
 'configure --enable-gcc-warnings=warn-only
 --enable-gtk-deprecation-warnings --without-pop --with-mailutils
 --enable-checking=all --enable-check-lisp-object-type --with-modules
 'CFLAGS=-O0 -ggdb3''

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP
SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LC_TIME: en_DK.utf8
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail phst skeleton pcase ffap
thingatpt url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs json map url-vars rx message mailcap
yank-media rmc dired dired-loaddefs rfc822 mml mml-sec password-cache
epa derived epg rfc6068 epg-config gnus-util time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader gnutls
puny elp dbus xml seq gv subr-x byte-opt bytecomp byte-compile cconv
compile text-property-search comint ansi-color ring cl-loaddefs cl-lib
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget keymap hashtable-print-readable backquote threads
dbusbind inotify dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 67433 8542)
 (symbols 48 8204 1)
 (strings 32 23684 1581)
 (string-bytes 1 761000)
 (vectors 16 15596)
 (vector-slots 8 208549 48737)
 (floats 8 28 30)
 (intervals 56 230 0)
 (buffers 992 11))

-- 
Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich.  Falls Sie diese fälschlicherweise erhalten haben
sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie
alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail
an die falsche Person gesendet wurde.

This e-mail is confidential.  If you received this communication by mistake,
please don’t forward it to anyone else, please erase all copies and
attachments, and please let me know that it has gone to the wrong person.

bug#54126: 29.0.50; C-x x g doesn't always correctly revert SSHFS files

[Michael Albinus]

Philipp Stephani <p.stephani2@gmail.com> writes:

Hi Philipp,

> At least on my system the following happens often, but not always:
>
> Create some file whose contents don't really matter.  In my case:
>
> $ cat /tmp/a.c
> int main(void) {
>   return 0;
> }
>
> Visit the file over SSHFS:
>
> $ emacs -Q /sshfs:localhost:/tmp/a.c
>
> Now, outside of Emacs, append something to the file:
>
> $ echo aaaaa >> /tmp/a.c
>
> Immediately after that, back in Emacs, hit C-x x g.  The new content
> isn't there.  Only after reverting the buffer a second time it appears.
> First I thought this was a timing/cache coherency issue, but even
> waiting for 10 seconds doesn't fix it in most cases.  Somewhat
> surprisingly, switching to a different buffer in between appears to make
> the problem go away (in some cases at least).

Looks like you are plagued by caching. revert-buffer reverts a file
only, if it is modified on disk. Tramps caches file attributes by
default for 10 seconds (see remote-file-name-inhibit-cache). Set this
value to t in order to test, whether it makes a difference. However, you
have said you did wait for 10 seconds, so maybe this isn't the reason.

Another cache might come from sshfs itself. Tramp calls sshfs like
"sshfs localhost:/ /tmp/tramp.sshfs.localhost -C -o idmap=user,reconnect".
See tramp-mount-args settings in tramp-sshfs.el, line 33-34. You might
try to add other options, like "-o no_readahead" or "-o sync_readdir",
see sshfs(1). Don't forget to unmount the sshfs mount point, before you
start a new Emacs session with changed options.

Best regards, Michael.

bug#54126: 29.0.50; C-x x g doesn't always correctly revert SSHFS files

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Philipp,

>> At least on my system the following happens often, but not always:
>>
>> Create some file whose contents don't really matter.  In my case:
>>
>> $ cat /tmp/a.c
>> int main(void) {
>>   return 0;
>> }
>>
>> Visit the file over SSHFS:
>>
>> $ emacs -Q /sshfs:localhost:/tmp/a.c
>>
>> Now, outside of Emacs, append something to the file:
>>
>> $ echo aaaaa >> /tmp/a.c
>>
>> Immediately after that, back in Emacs, hit C-x x g.  The new content
>> isn't there.  Only after reverting the buffer a second time it appears.
>> First I thought this was a timing/cache coherency issue, but even
>> waiting for 10 seconds doesn't fix it in most cases.  Somewhat
>> surprisingly, switching to a different buffer in between appears to make
>> the problem go away (in some cases at least).
>
> Looks like you are plagued by caching. revert-buffer reverts a file
> only, if it is modified on disk. Tramps caches file attributes by
> default for 10 seconds (see remote-file-name-inhibit-cache). Set this
> value to t in order to test, whether it makes a difference. However, you
> have said you did wait for 10 seconds, so maybe this isn't the reason.
>
> Another cache might come from sshfs itself. Tramp calls sshfs like
> "sshfs localhost:/ /tmp/tramp.sshfs.localhost -C -o idmap=user,reconnect".
> See tramp-mount-args settings in tramp-sshfs.el, line 33-34. You might
> try to add other options, like "-o no_readahead" or "-o sync_readdir",
> see sshfs(1). Don't forget to unmount the sshfs mount point, before you
> start a new Emacs session with changed options.

The appended patch fixes the problem for me. Since it disables the
directory cache of sshfs, there might be performance penalties. Could
you, pls, check how it behaves for you? Shall we enable this by default?

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 462 bytes --]

*** /tmp/ediffSUoOdM	2022-03-10 14:04:09.484978898 +0100
--- /home/albinus/src/tramp/lisp/tramp-sshfs.el	2022-03-10 14:01:22.869415240 +0100
***************
*** 51,56 ****
--- 51,57 ----
   (add-to-list 'tramp-methods
  	      `(,tramp-sshfs-method
  		(tramp-mount-args            (("-C") ("-p" "%p")
+ 					      ("-o" "dir_cache=no")
  					      ("-o" "transform_symlinks")
  					      ("-o" "idmap=user,reconnect")))
  		;; These are for remote processes.

bug#54126: 29.0.50; C-x x g doesn't always correctly revert SSHFS files

[Michael Albinus]

Version: 25.2

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Philipp,

> The appended patch fixes the problem for me. Since it disables the
> directory cache of sshfs, there might be performance penalties. Could
> you, pls, check how it behaves for you? Shall we enable this by default?

No further comment, so I've added this mount option in
tramp-sshfs.el. Pushed to the repositories. It will also be available
with the next Tramp GNU ELPA version 2.5.2.3.

Closing the bug.

Best regards, Michael.

bug#54126: 29.0.50; C-x x g doesn't always correctly revert SSHFS files

[Philipp Stephani]

Am Mi., 23. Feb. 2022 um 16:14 Uhr schrieb Michael Albinus
<michael.albinus@gmx.de>:
>
> Philipp Stephani <p.stephani2@gmail.com> writes:
>
> Hi Philipp,
>
> > At least on my system the following happens often, but not always:
> >
> > Create some file whose contents don't really matter.  In my case:
> >
> > $ cat /tmp/a.c
> > int main(void) {
> >   return 0;
> > }
> >
> > Visit the file over SSHFS:
> >
> > $ emacs -Q /sshfs:localhost:/tmp/a.c
> >
> > Now, outside of Emacs, append something to the file:
> >
> > $ echo aaaaa >> /tmp/a.c
> >
> > Immediately after that, back in Emacs, hit C-x x g.  The new content
> > isn't there.  Only after reverting the buffer a second time it appears.
> > First I thought this was a timing/cache coherency issue, but even
> > waiting for 10 seconds doesn't fix it in most cases.  Somewhat
> > surprisingly, switching to a different buffer in between appears to make
> > the problem go away (in some cases at least).
>
> Looks like you are plagued by caching. revert-buffer reverts a file
> only, if it is modified on disk. Tramps caches file attributes by
> default for 10 seconds (see remote-file-name-inhibit-cache). Set this
> value to t in order to test, whether it makes a difference. However, you
> have said you did wait for 10 seconds, so maybe this isn't the reason.

I didn't use a stopwatch :-)
So if the cache timeout is really exactly 10 seconds, then it's rather
likely I was affected by it.
This doesn't seem to happen with SSH, only with SSHFS, though. Does
the caching behavior differ? I.e., is the SSH backend stricter about
cache coherency?
It's also worth noting that Unix tools generally expect filesystems to
be sequentially consistent (i.e. any modification is immediately
visible by any other consumer, and there's a strict ordering between
operations). Caching is still possible, but it shouldn't break these
consistency guarantees.

>
> Another cache might come from sshfs itself. Tramp calls sshfs like
> "sshfs localhost:/ /tmp/tramp.sshfs.localhost -C -o idmap=user,reconnect".
> See tramp-mount-args settings in tramp-sshfs.el, line 33-34. You might
> try to add other options, like "-o no_readahead" or "-o sync_readdir",
> see sshfs(1). Don't forget to unmount the sshfs mount point, before you
> start a new Emacs session with changed options.

I'll play around with that, thanks for the hint.
If this turns out to be the problem, then caching should probably be
disabled by default, because sequential consistency is more important
than potential speed-ups.

bug#54126: 29.0.50; C-x x g doesn't always correctly revert SSHFS files

[Michael Albinus]

Philipp Stephani <p.stephani2@gmail.com> writes:

Hi Philipp,

>> Another cache might come from sshfs itself. Tramp calls sshfs like
>> "sshfs localhost:/ /tmp/tramp.sshfs.localhost -C -o idmap=user,reconnect".
>> See tramp-mount-args settings in tramp-sshfs.el, line 33-34. You might
>> try to add other options, like "-o no_readahead" or "-o sync_readdir",
>> see sshfs(1). Don't forget to unmount the sshfs mount point, before you
>> start a new Emacs session with changed options.
>
> I'll play around with that, thanks for the hint.
> If this turns out to be the problem, then caching should probably be
> disabled by default, because sequential consistency is more important
> than potential speed-ups.

This is fixed already in git master, see my other message. The trick was
to set "-o dir_cache=no", which disables the sshfs directory cache.

Best regards, Michael.