Issue with org-persist and Tramp

Issue with org-persist and Tramp

[Fabio Natali]

Hi,

I seem to be having an issue with org-persist and Tramp which is close
to what discussed in this thread:

https://lists.gnu.org/archive/html//emacs-orgmode/2022-05/msg00720.html

- I open a Org file on a remote machine, via Tramp.
- The SSH connection dies.
- I try to kill the stale buffer with kill-buffer and from IBuffer.
- The buffer is not killed, instead Emacs hangs for a while while still
  trying to connect to the old SSH connection.
- The buffer becomes sentient and gains immortality... either that or I
  restart Emacs, which is clearly a big no-no. :D

Looking at the backtrace, it looks like the kill operation insists on
calling org-persist-write-all-buffer, which in turn seems to be calling
Tramp and therefore SSH to the now unreachable machine.

More context:

- The SSH process is now gone, no trace of its original process.
- I've tried all combinations of tramp-cleanup-* commands, with no luck.
- I'm on Emacs 29.1 and Org 9.6.9.
- The default value of org-persist-remote-files seems to be 100.

Is there any org-persist-related setting that I should include in my
init.el? Anything that I might be missing here?

Thanks for your help, cheers, Fabio.

Re: Issue with org-persist and Tramp

[Antonio Carlos Padoan Junior]

Hello Fabio,

Fabio Natali <me@fabionatali.com> writes:

> Hi,
>
> I seem to be having an issue with org-persist and Tramp which is close
> to what discussed in this thread:
>
> https://lists.gnu.org/archive/html//emacs-orgmode/2022-05/msg00720.html
>
> - I open a Org file on a remote machine, via Tramp.
> - The SSH connection dies.
> - I try to kill the stale buffer with kill-buffer and from IBuffer.
> - The buffer is not killed, instead Emacs hangs for a while while still
>   trying to connect to the old SSH connection.
> - The buffer becomes sentient and gains immortality... either that or I

I was intrigued by the same issue. My workaround is to save the buffer
locally (perhaps in /tmp). This action releases the buffer. I think this
behavior is probably intended to avoid loosing "local" work after the ssh
connection dies, but who knows...  

Anyway, if there is any other workaround I would be interested as well.

BR,
-- 
Antonio Carlos PADOAN JUNIOR
GPG fingerprint:
243F 237F 2DD3 4DCA 4EA3  1341 2481 90F9 B421 A6C9

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-07, 22:08 +0100, Antonio Carlos Padoan Junior <acpadoanjr@yahoo.com.br> wrote:
> My workaround is to save the buffer locally (perhaps in /tmp). This
> action releases the buffer.

Hi Antonio,

Thanks for getting back to me.

Unfortunately your work-around doesn't seem to work in my case. Just to
make sure I'm not missing anything, when you say you save the buffer
locally, you mean with `write-file' (C-x C-w), right?

If I use `write-file', the system lags for a while, then it prompts me
to insert a file name. Whatever path I insert, Emacs will try to use the
old SSH connection and therefore freeze. If I `keyboard-quit' (C-g), the
system "unfreezes" but the file won't be saved.

Thanks, best wishes, Fabio.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

Fabio Natali <me@fabionatali.com> writes:

> I seem to be having an issue with org-persist and Tramp which is close
> to what discussed in this thread:
>
> https://lists.gnu.org/archive/html//emacs-orgmode/2022-05/msg00720.html

After that thread, Org should not (by default) examine remote files.

> - I open a Org file on a remote machine, via Tramp.
> - The SSH connection dies.
> - I try to kill the stale buffer with kill-buffer and from IBuffer.
> - The buffer is not killed, instead Emacs hangs for a while while still
>   trying to connect to the old SSH connection.
> - The buffer becomes sentient and gains immortality... either that or I
>   restart Emacs, which is clearly a big no-no. :D
>
> Looking at the backtrace, it looks like the kill operation insists on
> calling org-persist-write-all-buffer, which in turn seems to be calling
> Tramp and therefore SSH to the now unreachable machine.

May you please share the backtrace?
In particular, may you (1) M-x toggle-debug-on-quit (2) try to close the
problematic file; observe Emacs "freeze" (3) C-g and post the obtained
backtrace.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-08, 09:25 +0000, Ihor Radchenko <yantar92@posteo.net> wrote:
> May you please share the backtrace?
> In particular, may you (1) M-x toggle-debug-on-quit (2) try to close the
> problematic file; observe Emacs "freeze" (3) C-g and post the obtained
> backtrace.

Hi Ihor,

Thanks for your email. Sure, glad to share the backtrace with you.

This is how the issue can be reproduced on my system.

- Enable debug, `toggle-debug-on-quit'
- Open a remote file, `(find-file "/ssh:<remote-machine>:~/test.org")'
- Bring the remote machine offline
- Try to close the remote file, `kill-buffer'
- The system hangs for a while as it tries to reach the remote machine
- Quitting (`keyboard-quit') produces the following backtrace

#+begin_export ascii
Debugger entered--Lisp error: (quit "")
  signal(quit (""))
  tramp-error(nil quit "")
  tramp-signal-hook-function(quit (""))
  signal(quit (""))
  tramp-maybe-open-connection((tramp-file-name "ssh" nil nil "<remote-machine>" nil "/home/user/test.org" nil))
  tramp-send-command((tramp-file-name "ssh" nil nil "<remote-machine>" nil "/home/user/test.org" nil) "echo \\\"`getconf PATH 2>/dev/null`\\\" 2>/dev/null; e...")
  tramp-send-command-and-check((tramp-file-name "ssh" nil nil "<remote-machine>" nil "/home/user/test.org" nil) "echo \\\"`getconf PATH 2>/dev/null`\\\"")
  tramp-send-command-and-read((tramp-file-name "ssh" nil nil "<remote-machine>" nil "/home/user/test.org" nil) "echo \\\"`getconf PATH 2>/dev/null`\\\"" noerror)
  tramp-get-remote-path((tramp-file-name "ssh" nil nil "<remote-machine>" nil "/home/user/test.org" nil))
  tramp-get-remote-stat((tramp-file-name "ssh" nil nil "<remote-machine>" nil "/home/user/test.org" nil))
  tramp-sh-handle-file-attributes("/ssh:<remote-machine>:/home/user/test.org")
  tramp-sh-file-name-handler(file-attributes "/ssh:<remote-machine>:/home/user/test.org")
  apply(tramp-sh-file-name-handler file-attributes "/ssh:<remote-machine>:/home/user/test.org")
  tramp-file-name-handler(file-attributes "/ssh:<remote-machine>:/home/user/test.org")
  org-persist--normalize-associated(#<buffer test.org>)
  org-persist-write-all(#<buffer test.org>)
  org-persist-write-all-buffer()
  kill-buffer("test.org")
  funcall-interactively(kill-buffer "test.org")
  command-execute(kill-buffer)
#+end_export

Glad to share other info if helpful.

Thanks, cheers, Fabio.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

Fabio Natali <me@fabionatali.com> writes:

> Thanks for your email. Sure, glad to share the backtrace with you.
> ...
>   tramp-file-name-handler(file-attributes "/ssh:<remote-machine>:/home/user/test.org")
>   org-persist--normalize-associated(#<buffer test.org>)

I see the problem now.
Does the attached patch solve the "freeze"?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-persist-normalize-associated-Avoid-TRAMP-connect.patch --]
[-- Type: text/x-patch, Size: 1741 bytes --]

From ac571f9654ef5de8cef7157e216beeb0b91f6125 Mon Sep 17 00:00:00 2001
Message-ID: <ac571f9654ef5de8cef7157e216beeb0b91f6125.1699466416.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 8 Nov 2023 19:58:42 +0200
Subject: [PATCH] org-persist--normalize-associated: Avoid TRAMP connection for
 remote files

* lisp/org-persist.el (org-persist--normalize-associated): Never try
to store inode association for remote TRAMP files.

Reported-by: Fabio Natali <me@fabionatali.com>
Link: https://orgmode.org/list/87jzqthdge.fsf@fabionatali.com
---
 lisp/org-persist.el | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/lisp/org-persist.el b/lisp/org-persist.el
index 01078f459..f97e1d7a4 100644
--- a/lisp/org-persist.el
+++ b/lisp/org-persist.el
@@ -481,9 +481,14 @@ (defun org-persist--normalize-associated (associated)
      (unless (stringp associated)
        (setq associated (cadr associated)))
      (let* ((rtn `(:file ,associated))
-            (inode (and (fboundp 'file-attribute-inode-number)
-                        (file-attribute-inode-number
-                         (file-attributes associated)))))
+            (inode (and
+                    ;; Do not store :inode for remote files - it may
+                    ;; be time-consuming on slow connections or even
+                    ;; fail completely when ssh connection is closed.
+                    (not (file-remote-p associated))
+                    (fboundp 'file-attribute-inode-number)
+                    (file-attribute-inode-number
+                     (file-attributes associated)))))
        (when inode (plist-put rtn :inode inode))
        rtn))
     ((or (pred bufferp) `(:buffer ,_))
-- 
2.42.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-08, 18:01 +0000, Ihor Radchenko <yantar92@posteo.net> wrote:
> I see the problem now.
> Does the attached patch solve the "freeze"?

Hey, thanks for sending this, really appreciate it.

Unfortunately that didn't seem to fix it though. Here's what I did.

- Added `(not (file-remote-p associated))' to the right place
- Revaluated the function `org-persist--normalize-associated'
- Tried to close the stale buffer (without repeating the entire
  workflow, just trying with the buffer from the previous test)

Result: the problem is still there, same backtrace.

If anything else comes to mind, happy to test it.

Cheers, Fabio.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

Fabio Natali <me@fabionatali.com> writes:

> On 2023-11-08, 18:01 +0000, Ihor Radchenko <yantar92@posteo.net> wrote:
>> I see the problem now.
>> Does the attached patch solve the "freeze"?
>
> Hey, thanks for sending this, really appreciate it.
>
> Unfortunately that didn't seem to fix it though. Here's what I did.
>
> - Added `(not (file-remote-p associated))' to the right place
> - Revaluated the function `org-persist--normalize-associated'
> - Tried to close the stale buffer (without repeating the entire
>   workflow, just trying with the buffer from the previous test)
>
> Result: the problem is still there, same backtrace.

Then I need to ask about
tramp-file-name-handler(file-attributes "/ssh:<remote-machine>:/home/user/test.org")

"/ssh:<remote-machine>:/home/user/test.org" is obfuscated to mask your
personal data, but it apparently fails to be filtered by `file-remote-p'
check. May I know what
   (file-remote-p "/ssh:<remote-machine>:/home/user/test.org")
(with appropriate <remote-machine>) returns on your side?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-09, 12:17 +0000, Ihor Radchenko <yantar92@posteo.net> wrote:
> May I know what
>    (file-remote-p "/ssh:<remote-machine>:/home/user/test.org")
> (with appropriate <remote-machine>) returns on your side?

Hi Ihor,

Sure, no particular reason to obfuscate the remote address, it's
actually a LAN IP. The above function returns "/ssh:192.168.1.100:". Is
that the kind of output one should be expecting? Anything wrong about
it?

Thanks, cheers, Fabio.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 578 bytes --]

Fabio Natali <me@fabionatali.com> writes:

> On 2023-11-09, 12:17 +0000, Ihor Radchenko <yantar92@posteo.net> wrote:
>> May I know what
>>    (file-remote-p "/ssh:<remote-machine>:/home/user/test.org")
>> (with appropriate <remote-machine>) returns on your side?
>
> Hi Ihor,
>
> Sure, no particular reason to obfuscate the remote address, it's
> actually a LAN IP. The above function returns "/ssh:192.168.1.100:". Is
> that the kind of output one should be expecting? Anything wrong about
> it?

Nothing wrong. Just me missing one code branch.
What about the attached patch?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-org-persist-normalize-associated-Avoid-TRAMP-conn.patch --]
[-- Type: text/x-patch, Size: 2450 bytes --]

From 73e2fc687b966f3c12b87dd40c90cdc6b9f440fc Mon Sep 17 00:00:00 2001
Message-ID: <73e2fc687b966f3c12b87dd40c90cdc6b9f440fc.1699608346.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 8 Nov 2023 19:58:42 +0200
Subject: [PATCH v2] org-persist--normalize-associated: Avoid TRAMP connection
 for remote files

* lisp/org-persist.el (org-persist--normalize-associated): Never try
to store inode association for remote TRAMP files.

Reported-by: Fabio Natali <me@fabionatali.com>
Link: https://orgmode.org/list/87jzqthdge.fsf@fabionatali.com
---
 lisp/org-persist.el | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/lisp/org-persist.el b/lisp/org-persist.el
index 81edcf6f2..bdc6006b6 100644
--- a/lisp/org-persist.el
+++ b/lisp/org-persist.el
@@ -614,9 +614,14 @@ (defun org-persist--normalize-associated (associated)
      (unless (stringp associated)
        (setq associated (cadr associated)))
      (let* ((rtn `(:file ,associated))
-            (inode (and (fboundp 'file-attribute-inode-number)
-                        (file-attribute-inode-number
-                         (file-attributes associated)))))
+            (inode (and
+                    ;; Do not store :inode for remote files - it may
+                    ;; be time-consuming on slow connections or even
+                    ;; fail completely when ssh connection is closed.
+                    (not (file-remote-p associated))
+                    (fboundp 'file-attribute-inode-number)
+                    (file-attribute-inode-number
+                     (file-attributes associated)))))
        (when inode (plist-put rtn :inode inode))
        rtn))
     ((or (pred bufferp) `(:buffer ,_))
@@ -634,6 +639,10 @@ (defun org-persist--normalize-associated (associated)
                      (or (buffer-base-buffer associated)
                          associated)))
          (setq inode (when (and file
+                                ;; Do not store :inode for remote files - it may
+                                ;; be time-consuming on slow connections or even
+                                ;; fail completely when ssh connection is closed.
+                                (not (file-remote-p associated))
                                 (fboundp 'file-attribute-inode-number))
                        (file-attribute-inode-number
                         (file-attributes file))))
-- 
2.42.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-10, 09:26 +0000, Ihor Radchenko <yantar92@posteo.net> wrote:
> What about the attached patch?

Hi Ihor,

Brilliant. I think the second branch should read `(not (file-remote-p
file))', i.e. `file' instead of `associated'? It seems to work with that
micro amendment.

Anything else I should be testing or does this look reasonable enough to
be pushed to the repo at some point?

Thank you very much for this!

🙏🙏🙏

Cheers, Fabio.

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-10, 11:13 +0000, Fabio Natali <me@fabionatali.com> wrote:
> Brilliant. I think the second branch should read `(not (file-remote-p
> file))', i.e. `file' instead of `associated'? It seems to work with that
> micro amendment.

Hm, I stand corrected, I retested it end-to-end and no, it doesn't seem
to work. I'll post more details later. Cheers, Fabio.

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-10, 11:26 +0000, Fabio Natali <me@fabionatali.com> wrote:
> On 2023-11-10, 11:13 +0000, Fabio Natali <me@fabionatali.com> wrote:
>> Brilliant. I think the second branch should read `(not (file-remote-p
>> file))', i.e. `file' instead of `associated'? It seems to work with that
>> micro amendment.
>
> Hm, I stand corrected, I retested it end-to-end and no, it doesn't seem
> to work. I'll post more details later. Cheers, Fabio.

Hi Ihor,

Sorry for the delay. My results below, after testing a couple of
different cases.

If the remote file had been saved before losing connection, then the
patch below will fix the problem. TRAMP will still hang for a while, as
it tries to reach the server but the buffer will be closed immediately
when using `tramp-cleanup-all-buffers' (or a variant thereof).

--- /tmp/test-0.el	2023-11-11 09:27:33.477117451 +0000
+++ /tmp/test-1.el	2023-11-11 09:27:14.217129703 +0000
@@ -5,7 +5,8 @@
      (unless (stringp associated)
        (setq associated (cadr associated)))
      (let* ((rtn `(:file ,associated))
-            (inode (and (fboundp 'file-attribute-inode-number)
+            (inode (and (not (file-remote-p associated))
+			(fboundp 'file-attribute-inode-number)
                         (file-attribute-inode-number
                          (file-attributes associated)))))
        (when inode (plist-put rtn :inode inode))
@@ -25,6 +26,7 @@
                      (or (buffer-base-buffer associated)
                          associated)))
          (setq inode (when (and file
+				(not (file-remote-p file))
                                 (fboundp 'file-attribute-inode-number))
                        (file-attribute-inode-number
                         (file-attributes file))))

(Apologies, the patch is badly formatted and not meant for the
repository.) The gist is that I replaced an occurrence of `associated'
with `file' in your latest patch, Ihor. It seems to work, what do you
think?

However, if the remote file hadn't been saved, then TRAMP will still
hang indefinitely. Quitting will reveal the following backtrace:

Debugger entered--Lisp error: (quit "")
  signal(quit (""))
  tramp-error(nil quit "")
  tramp-signal-hook-function(quit (""))
  signal(quit (""))
  tramp-maybe-open-connection((tramp-file-name "ssh" nil nil "192.168.1.100" nil "~/.#test.org" nil))
  tramp-send-command((tramp-file-name "ssh" nil nil "192.168.1.100" nil "~/.#test.org" nil) "echo ~ 2>/dev/null; echo tramp_exit_status $?")
  tramp-send-command-and-check((tramp-file-name "ssh" nil nil "192.168.1.100" nil "~/.#test.org" nil) "echo ~")
  tramp-sh-handle-get-home-directory((tramp-file-name "ssh" nil nil "192.168.1.100" nil "~/.#test.org" nil) "")
  tramp-sh-file-name-handler(tramp-get-home-directory (tramp-file-name "ssh" nil nil "192.168.1.100" nil "~/.#test.org" nil) "")
  apply(tramp-sh-file-name-handler tramp-get-home-directory ((tramp-file-name "ssh" nil nil "192.168.1.100" nil "~/.#test.org" nil) ""))
  tramp-file-name-handler(tramp-get-home-directory (tramp-file-name "ssh" nil nil "192.168.1.100" nil "~/.#test.org" nil) "")
  tramp-get-home-directory((tramp-file-name "ssh" nil nil "192.168.1.100" nil "~/.#test.org" nil) "")
  tramp-sh-handle-expand-file-name("/ssh:192.168.1.100:~/.#test.org" nil)
  tramp-sh-file-name-handler(expand-file-name "/ssh:192.168.1.100:~/.#test.org" nil)
  apply(tramp-sh-file-name-handler expand-file-name ("/ssh:192.168.1.100:~/.#test.org" nil))
  tramp-file-name-handler(expand-file-name "/ssh:192.168.1.100:~/.#test.org" nil)
  files--transform-file-name("/ssh:192.168.1.100:~/test.org" nil ".#" "")
  make-lock-file-name("/ssh:192.168.1.100:~/test.org")
  tramp-run-real-handler(make-lock-file-name ("/ssh:192.168.1.100:~/test.org"))
  tramp-handle-make-lock-file-name("/ssh:192.168.1.100:~/test.org")
  tramp-sh-file-name-handler(make-lock-file-name "/ssh:192.168.1.100:~/test.org")
  apply(tramp-sh-file-name-handler make-lock-file-name "/ssh:192.168.1.100:~/test.org")
  tramp-file-name-handler(make-lock-file-name "/ssh:192.168.1.100:~/test.org")
  tramp-compat-make-lock-file-name("/ssh:192.168.1.100:~/test.org")
  tramp-handle-unlock-file("/ssh:192.168.1.100:~/test.org")
  tramp-sh-file-name-handler(unlock-file "/ssh:192.168.1.100:~/test.org")
  apply(tramp-sh-file-name-handler unlock-file "/ssh:192.168.1.100:~/test.org")
  tramp-file-name-handler(unlock-file "/ssh:192.168.1.100:~/test.org")
  kill-buffer("test.org")
  funcall-interactively(kill-buffer "test.org")
  command-execute(kill-buffer)

This seems to be on the TRAMP side?

Cheers, Fabio.

Re: Issue with org-persist and Tramp

[Ihor Radchenko]

Fabio Natali <me@fabionatali.com> writes:

> Sorry for the delay. My results below, after testing a couple of
> different cases.
>
> If the remote file had been saved before losing connection, then the
> patch below will fix the problem. TRAMP will still hang for a while, as
> it tries to reach the server but the buffer will be closed immediately
> when using `tramp-cleanup-all-buffers' (or a variant thereof).
> ...
> (Apologies, the patch is badly formatted and not meant for the
> repository.) The gist is that I replaced an occurrence of `associated'
> with `file' in your latest patch, Ihor. It seems to work, what do you
> think?

I made the same change in my v2 patch.
Applied.
Fixed, on bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=866e638c2

> However, if the remote file hadn't been saved, then TRAMP will still
> hang indefinitely. Quitting will reveal the following backtrace:
>
> Debugger entered--Lisp error: (quit "")
> ...
>   tramp-file-name-handler(unlock-file "/ssh:192.168.1.100:~/test.org")
>   kill-buffer("test.org")
>   funcall-interactively(kill-buffer "test.org")
>   command-execute(kill-buffer)
>
> This seems to be on the TRAMP side?

Yes.
You may consider customizing `remote-file-name-inhibit-locks'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Issue with org-persist and Tramp

[Fabio Natali]

On 2023-11-11, 11:06 +0000, Ihor Radchenko <yantar92@posteo.net> wrote:
> Fixed, on bugfix.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=866e638c2

Super, thanks for helping with this, Ihor. 🙏

> You may consider customizing `remote-file-name-inhibit-locks'.

Ah! Good to know, thanks.

Cheers, F.