bug#64401: 28.1; Desktop restoration

bug#64401: 28.1; Desktop restoration

[Tom Hunt]

Often desktop files contain remote (TRAMP) buffers. Such buffers might
originally be open on hosts that are not currently accessible for
whatever reason.

When opening a desktop file, either automatically on startup or via
desktop-read, emacs will hang on trying to open such files. If the hang
is aborted via C-g, the entire desktop load will be aborted and other
buffers (which might still be accessible) are not opened.

Ideally, buffers requiring connections to unreachable hosts should be
skipped and the remainder of the desktop session opened.


In GNU Emacs 28.1 (build 1, x86_64-redhat-linux-gnu, GTK+ Version 3.24.34, cairo version 1.17.6)
 of 2022-07-20 built on buildvm-x86-04.iad2.fedoraproject.org
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Fedora Linux 37 (MATE-Compiz)

Configured using:
 'configure --build=x86_64-redhat-linux-gnu
 --host=x86_64-redhat-linux-gnu --program-prefix=
 --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
 --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
 --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png
 --with-rsvg --with-tiff --with-xpm --with-x-toolkit=gtk3 --with-gpm=no
 --with-xwidgets --with-modules --with-harfbuzz --with-cairo --with-json
 --with-native-compilation build_alias=x86_64-redhat-linux-gnu
 host_alias=x86_64-redhat-linux-gnu CC=gcc 'CFLAGS=-DMAIL_USE_LOCKF -O2
 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches
 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
 -Wp,-D_GLIBCXX_ASSERTIONS
 -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong
 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
 -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-z,relro
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XDBE XIM XPM XWIDGETS GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Org

Minor modes in effect:
  desktop-save-mode: t
  global-flycheck-mode: t
  pyvenv-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  auto-fill-function: org-auto-fill-function
  transient-mark-mode: t

Load-path shadows:
/home/tom/.emacs.d/elpa/transient-20230601.1854/transient hides /usr/share/emacs/28.1/lisp/transient

Features:
(shadow sort mail-extr emacsbug sendmail js imenu cc-mode cc-fonts
cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
cl-print shortdoc misearch multi-isearch mule-util tramp-cmds dired-aux
autorevert filenotify slime apropos arc-mode archive-mode hyperspec
sh-script smie executable vc-hg vc-git vc-bzr vc-dispatcher rfc2104
tramp-cache conf-mode org-element avl-tree ol-eww eww xdg url-queue
mm-url ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect gnus-search
eieio-opt speedbar ezimage dframe gnus-art mm-uu mml2015 mm-view
mml-smime smime dig gnus-sum shr kinsoku svg dom gnus-group gnus-undo
gnus-start gnus-dbus dbus gnus-cloud nnimap nnmail mail-source utf7
netrc nnoo gnus-spec gnus-int gnus-range message rmc puny rfc822 mml
mml-sec epa derived epg rfc6068 epg-config mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader gnus-win gnus
nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
mail-utils mm-util mail-prsvr ol-docview doc-view jka-compr image-mode
exif dired dired-loaddefs ol-bibtex ol-bbdb ol-w3m ol-doi org-link-doi
org org-macro org-footnote org-pcomplete org-list org-faces org-entities
noutline outline org-version ob-python ob ob-tangle org-src ob-ref
ob-lob ob-table ob-exp ob-comint ob-core ob-eval org-table oc-basic
bibtex ol org-keys oc org-loaddefs cal-menu calendar cal-loaddefs
desktop frameset web-mode disp-table org-crypt org-compat org-macs
flycheck find-func dash cl-extra yasnippet highlight-indentation
flymake-proc flymake warnings thingatpt company-capf company help-fns
radix-tree help-mode elpy advice edmacro kmacro elpy-rpc pyvenv eshell
esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg esh-module esh-groups
esh-util elpy-shell elpy-profile elpy-django s elpy-refactor diff-mode
easy-mmode python tramp-sh tramp tramp-loaddefs trampver
tramp-integration tramp-compat shell pcomplete parse-time iso8601
time-date ls-lisp format-spec hideshow grep compile text-property-search
comint ansi-color files-x etags fileloop generator xref project ring
cus-edit pp cus-load wid-edit ido finder-inf clang-rename
clang-include-fixer let-alist clang-format xml rx pcase slime-autoloads
info package browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads xwidget-internal dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 673112 31758)
 (symbols 48 41109 5)
 (strings 32 213884 4480)
 (string-bytes 1 6311007)
 (vectors 16 67995)
 (vector-slots 8 1165607 55871)
 (floats 8 439 83)
 (intervals 56 1472 168)
 (buffers 992 24))

bug#64401: 28.1; Desktop restoration

[Eli Zaretskii]

> Date: Fri, 30 Jun 2023 17:52:02 +0000
> From: Tom Hunt <tom@tomhunt.email>
> 
> 
> Often desktop files contain remote (TRAMP) buffers. Such buffers might
> originally be open on hosts that are not currently accessible for
> whatever reason.
> 
> When opening a desktop file, either automatically on startup or via
> desktop-read, emacs will hang on trying to open such files. If the hang
> is aborted via C-g, the entire desktop load will be aborted and other
> buffers (which might still be accessible) are not opened.
> 
> Ideally, buffers requiring connections to unreachable hosts should be
> skipped and the remainder of the desktop session opened.

desktop.el already tries to do that, see desktop-files-not-to-save.
Any idea why it doesn't work in your case?

bug#64401: 28.1; Desktop restoration

[Tom Hunt]

> 
> 
> desktop.el already tries to do that, see desktop-files-not-to-save.
> Any idea why it doesn't work in your case?

I've set desktop-files-not-to-save to nil. This is because I explicitly want remote files saved, most of the time. (Much of my emacs workflow involves remote files; if they aren't saved then desktop saving is substantially less useful.)

The problem isn't at save time, it's at restore time. The issue is that, if a saved remote file isn't reachable, it hangs the whole restore. Ideally, an unreachable saved file would just be skipped over (at restore time) and the rest of the session restored.

bug#64401: 28.1; Desktop restoration

[Eli Zaretskii]

> Date: Sat, 01 Jul 2023 18:46:54 +0000
> From: Tom Hunt <tom@tomhunt.email>
> Cc: 64401@debbugs.gnu.org
> 
> > desktop.el already tries to do that, see desktop-files-not-to-save.
> > Any idea why it doesn't work in your case?
> 
> I've set desktop-files-not-to-save to nil. This is because I explicitly want remote files saved, most of the time. (Much of my emacs workflow involves remote files; if they aren't saved then desktop saving is substantially less useful.)
> 
> The problem isn't at save time, it's at restore time. The issue is that, if a saved remote file isn't reachable, it hangs the whole restore. Ideally, an unreachable saved file would just be skipped over (at restore time) and the rest of the session restored.

I don't understand how this could be done, given that discovery of
inaccessible files is done by trying to access them.  If you can think
about design of this, maybe we could implement such a feature.

Michael, does Tramp have optional behavior which would abandon attempt
to visit a file if it takes more than some predefined time?

bug#64401: 28.1; Desktop restoration

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> The problem isn't at save time, it's at restore time. The issue is
>> that, if a saved remote file isn't reachable, it hangs the whole
>> restore. Ideally, an unreachable saved file would just be skipped
>> over (at restore time) and the rest of the session restored.
>
> I don't understand how this could be done, given that discovery of
> inaccessible files is done by trying to access them.  If you can think
> about design of this, maybe we could implement such a feature.
>
> Michael, does Tramp have optional behavior which would abandon attempt
> to visit a file if it takes more than some predefined time?

No, there isn't. And we cannot implement it for visiting files, because
we don't know what would be an acceptable time period for visiting. There
might be extremely huge files people try to visit over a slow connection.

But what about extending the semantics of `access-file'? It should
timeout after a given predefined time. The problem is how to find out
this timeout value. For remote files, it depends on the quality of
connection (and perhaps on the performance of the remote machine / file
system), so we must make it configurable, with reasonable defaults.

This timeout might be even useful for local files. Think about hanging
mount points and alike.

Best regards, Michael.

bug#64401: 28.1; Desktop restoration

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Tom Hunt <tom@tomhunt.email>,  64401@debbugs.gnu.org
> Date: Sun, 02 Jul 2023 10:29:37 +0200
> 
> > Michael, does Tramp have optional behavior which would abandon attempt
> > to visit a file if it takes more than some predefined time?
> 
> No, there isn't. And we cannot implement it for visiting files, because
> we don't know what would be an acceptable time period for visiting. There
> might be extremely huge files people try to visit over a slow connection.
> 
> But what about extending the semantics of `access-file'? It should
> timeout after a given predefined time. The problem is how to find out
> this timeout value. For remote files, it depends on the quality of
> connection (and perhaps on the performance of the remote machine / file
> system), so we must make it configurable, with reasonable defaults.

Something like that would be useful, yes.

> This timeout might be even useful for local files. Think about hanging
> mount points and alike.

That could be trickier to implement, since AFAIR we call C APIs for
that.

bug#64401: 28.1; Desktop restoration

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> From: Michael Albinus <michael.albinus@gmx.de>
>> Cc: Tom Hunt <tom@tomhunt.email>,  64401@debbugs.gnu.org
>> Date: Sun, 02 Jul 2023 10:29:37 +0200
>>
>> > Michael, does Tramp have optional behavior which would abandon attempt
>> > to visit a file if it takes more than some predefined time?
>>
>> No, there isn't. And we cannot implement it for visiting files, because
>> we don't know what would be an acceptable time period for visiting. There
>> might be extremely huge files people try to visit over a slow connection.
>>
>> But what about extending the semantics of `access-file'? It should
>> timeout after a given predefined time. The problem is how to find out
>> this timeout value. For remote files, it depends on the quality of
>> connection (and perhaps on the performance of the remote machine / file
>> system), so we must make it configurable, with reasonable defaults.
>
> Something like that would be useful, yes.

What about

--8<---------------cut here---------------start------------->8---
(defcustom remote-file-name-access-timeout 30
  "Timeout (in seconds) for `access-file'.
This timeout limits the time to check, whether a remote file is
accessible.  `access-file' returns an error after that time.  If
the value is nil, no timeout is used.

For slow connections, it might be useful to increase the value."
  :group 'files
  :version "30.1"
  :type '(choice :tag "Timeout (seconds)" natnum (const nil)))
--8<---------------cut here---------------end--------------->8---

Implementation note: This would be the timeout if the connection is
already established. If there is no connection yet, Tramp would add that
initialization time. Establishing a new connection is always limited by
a reasonable timeout.

>> This timeout might be even useful for local files. Think about hanging
>> mount points and alike.
>
> That could be trickier to implement, since AFAIR we call C APIs for
> that.

OK, so we don't do it.

Best regards, Michael.

bug#64401: 28.1; Desktop restoration

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: tom@tomhunt.email,  64401@debbugs.gnu.org
> Date: Sun, 02 Jul 2023 11:23:10 +0200
> 
> What about
> 
> --8<---------------cut here---------------start------------->8---
> (defcustom remote-file-name-access-timeout 30
>   "Timeout (in seconds) for `access-file'.
> This timeout limits the time to check, whether a remote file is
> accessible.  `access-file' returns an error after that time.  If
> the value is nil, no timeout is used.
> 
> For slow connections, it might be useful to increase the value."
>   :group 'files
>   :version "30.1"
>   :type '(choice :tag "Timeout (seconds)" natnum (const nil)))
> --8<---------------cut here---------------end--------------->8---

Sounds good.  Regarding the default value: how long does it take for
access-file to do its job with "normal" connections?  Maybe 30 is too
long, and something like 10 will be better?

Also, the implementation should probably "remember" the result and
apply it to all the other files on the same volume?

> Implementation note: This would be the timeout if the connection is
> already established. If there is no connection yet, Tramp would add that
> initialization time. Establishing a new connection is always limited by
> a reasonable timeout.

You mean, we already have a timeout for new connections?  If so, why
didn't the OP see its effect?

Thanks.

bug#64401: 28.1; Desktop restoration

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> Sounds good.  Regarding the default value: how long does it take for
> access-file to do its job with "normal" connections?  Maybe 30 is too
> long, and something like 10 will be better?

OK, let's start with this.

> Also, the implementation should probably "remember" the result and
> apply it to all the other files on the same volume?

Hmm, this will be harder. `access-file' could fail for different
reasons. And Tramp does not know whether two remote files belong to the
same volume. Well, there is `file-attribute-device-number', but this is
not trustworthy for all different kind of remote file systems.

I'll see whether I can implement a cache for different volumes on remote
hosts where it makes sense.

>> Implementation note: This would be the timeout if the connection is
>> already established. If there is no connection yet, Tramp would add that
>> initialization time. Establishing a new connection is always limited by
>> a reasonable timeout.
>
> You mean, we already have a timeout for new connections?  If so, why
> didn't the OP see its effect?

Don't know, I would need the traces. Tom, could you please start "emacs
--eval '(setq tramp-verbose 10)'" and reproduce the problem? There will
be a Tramp debug buffer, which you could show us.

> Thanks.

Best regards, Michael.

bug#64401: 28.1; Desktop restoration

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Eli,

>> Also, the implementation should probably "remember" the result and
>> apply it to all the other files on the same volume?
>
> Hmm, this will be harder. `access-file' could fail for different
> reasons. And Tramp does not know whether two remote files belong to the
> same volume. Well, there is `file-attribute-device-number', but this is
> not trustworthy for all different kind of remote file systems.
>
> I'll see whether I can implement a cache for different volumes on remote
> hosts where it makes sense.

Going through the Tramp implementation, there's no need for such a
cache. The usual Tramp cache does it already sufficiently.

Either a file has already cached information, then that's fine. Or it
has no cache information, then we would need to call file-attributes
first in order to get the device number, and then it is in the cache.

Finally, I've pushed a fix to master along the idea sketched in this
thread. I've decided to set the default of remote-file-name-access-timeout
to nil (which preserves the existing behavior). There are so many
different possibilities I couldn't decide for a sensitive value.

The Tramp manual explains now

--8<---------------cut here---------------start------------->8---
     Some packages, like ‘desktop.el’ or ‘recentf.el’, access remote
     files when loaded.  If the respective file is not accessible, TRAMP
     could block.  In order to check whether this could happen, add a
     test via ‘access-file’ with a proper timeout prior loading these
     packages:

          (let ((remote-file-name-access-timeout 10))
            (access-file "/method:user@host:/path/to/file" "error"))
          ⇒ nil

     The result ‘nil’ means success.  If the file is not accessible, or
     if the underlying operations last too long, ‘access-file’ returns
     with an error.

     The value of the timeout (10 seconds in the example) depends on
     your preference and on the quality of the connection to the remote
     host.  If the connection to the remote host isn’t established yet,
     and if this requires an interactive password, the timeout check
     doesn’t work properly.
--8<---------------cut here---------------end--------------->8---

Better would be, if desktop.el and recentf.el do this check on their
own. recentf.el uses file-readable-p (in recentf-keep-default-predicate).

desktop.el usually skips Tramp files (in desktop-files-not-to-save), but
this could be changed. In general it uses file-exists-p (in
desktop-restore-file-buffer) for a check before restoring.

Both implementations should be changed to use access-file instead. Will
try it next days. Do you know other packages in vanilla Emacs which
could profit from this mechanism?

And then we would need a good place to train users of desktop.el and
recentf.el to adapt remote-file-name-access-timeout - I have no idea
where.

>> Thanks.

Best regards, Michael.

bug#64401: 28.1; Desktop restoration

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

> Both implementations should be changed to use access-file instead. Will
> try it next days. Do you know other packages in vanilla Emacs which
> could profit from this mechanism?

I've pushed a patch to master implementing this n desktop.el and recentf.el.

> And then we would need a good place to train users of desktop.el and
> recentf.el to adapt remote-file-name-access-timeout - I have no idea
> where.

Well, I've added it in the Emacs manual. Where else ...

Tom, do you have a chance to test the Emacs master branch from git?

Otherwise, there's nothing else to do I believe.

Best regards, Michael.

bug#64401: 28.1; Desktop restoration

[Michael Albinus]

Version: 30.1

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

>> Both implementations should be changed to use access-file instead. Will
>> try it next days. Do you know other packages in vanilla Emacs which
>> could profit from this mechanism?
>
> I've pushed a patch to master implementing this n desktop.el and recentf.el.
>
>> And then we would need a good place to train users of desktop.el and
>> recentf.el to adapt remote-file-name-access-timeout - I have no idea
>> where.
>
> Well, I've added it in the Emacs manual. Where else ...
>
> Tom, do you have a chance to test the Emacs master branch from git?
>
> Otherwise, there's nothing else to do I believe.

No further comment, so I'm closing the bug.

Best regards, Michael.