Killing a hung ssh process in a TRAMP session

Killing a hung ssh process in a TRAMP session

[Suvayu Ali]

Hi everyone,

I have a small problem. I usually have to develop code remotely. 
(usually across the Atlantic :) ) The connection is not always very 
stable. Whenever there are any network problems, TRAMP keeps trying to 
reconnect to the remote machine.

My problem is whenever TRAMP is trying to reconnect, the whole emacs 
session hangs. The only way out of this situation is to wait for the 
connection to be established (not always practical as often it might 
take over 10 mins for the network to be back to normal) or kill emacs 
with `kill -9'. And since I usually run emacs in server mode, this means 
loosing a really complicated session with lots of buffers on several 
remote machines.

Is there any way I could tell emacs not to reconnect when this happens? 
I tried running `ps uf -u `whoami`' and kill the shell running the hung 
ssh process. But that only re-spawns the process again.

Is there anyway around this? Hopefully I have explained my predicament 
clearly. Thanks for any advice on this.

-- 
Suvayu

Open source is the future. It sets us free.

Re: Killing a hung ssh process in a TRAMP session

[Michael Albinus]

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> Hi everyone,

Hi,

> Is there any way I could tell emacs not to reconnect when this
> happens? I tried running `ps uf -u `whoami`' and kill the shell
> running the hung ssh process. But that only re-spawns the process
> again.
>
> Is there anyway around this? Hopefully I have explained my predicament
> clearly. Thanks for any advice on this.

Usually, you should be able to interrupt Tramp's activities by
C-g. Which version of Tramp do you use? You find it out by
"C-h v tramp-version".

Furthermore, there are some commands to close a connection.
"M-x tramp-cleanup-connection" allows you to close a connection.
"M-x tramp-cleanup-all-connections" closes all existing connections, and
"M-x tramp-cleanup-all-buffers" does the same, and deletes also related
buffers.

Best regards, Michael.

Re: Killing a hung ssh process in a TRAMP session

[Suvayu Ali]

Hi Michael,

On Thursday 07 January 2010 12:41 AM, Michael Albinus wrote:
> Suvayu Ali<fatkasuvayu+linux@gmail.com>  writes:
>
>> Is there any way I could tell emacs not to reconnect when this
>> happens? I tried running `ps uf -u `whoami`' and kill the shell
>> running the hung ssh process. But that only re-spawns the process
>> again.
>>
>> Is there anyway around this? Hopefully I have explained my predicament
>> clearly. Thanks for any advice on this.
>
> Usually, you should be able to interrupt Tramp's activities by
> C-g. Which version of Tramp do you use? You find it out by
> "C-h v tramp-version".
>

My version of tramp is 2.1.15, Emacs 23.1.1.

> Furthermore, there are some commands to close a connection.
> "M-x tramp-cleanup-connection" allows you to close a connection.
> "M-x tramp-cleanup-all-connections" closes all existing connections, and
> "M-x tramp-cleanup-all-buffers" does the same, and deletes also related
> buffers.
>

These will be a lot of help when emacs is still responsive, thanks. But 
usually when this happens, all the frames associated to the emacs server 
freezes, and doesn't respond to any keyboard or mouse input. I also see 
similar behaviour whenever TRAMP is busy with the network, say 
authenticating to the remote server. As soon as its done, every thing 
goes back to normal. Maybe I should have taken screen shots. :-\

> Best regards, Michael.

Thanks a lot for the response, I will try to take some screen shots the 
next time this happens.
-- 
Suvayu

Open source is the future. It sets us free.

Re: Killing a hung ssh process in a TRAMP session

[Michael Albinus]

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> These will be a lot of help when emacs is still responsive,
> thanks. But usually when this happens, all the frames associated to
> the emacs server freezes, and doesn't respond to any keyboard or mouse
> input. I also see similar behaviour whenever TRAMP is busy with the
> network, say authenticating to the remote server. As soon as its done,
> every thing goes back to normal. Maybe I should have taken screen
> shots. :-\

Please apply

  (setq debug-on-error t
        debug-on-signal t
        debug-on-quit t)

before the next test. Then, when Emacs hangs, you could try from a shell

# kill -FPE <emacspid>

Don't ask me why, but sometimes Emacs comes back with the FPE signal,
and you get a backtrace. This I would like to see, because then I could
protect that code with-local-quit and friends.

Best regards, Michael.

Re: Killing a hung ssh process in a TRAMP session

[Suvayu Ali]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

Hi Michael,

Sorry for the delay in my response, got a little busy with work.

On Thursday 07 January 2010 07:46 AM, Michael Albinus wrote:
> Suvayu Ali<fatkasuvayu+linux@gmail.com>  writes:
>
>> These will be a lot of help when emacs is still responsive,
>> thanks. But usually when this happens, all the frames associated to
>> the emacs server freezes, and doesn't respond to any keyboard or mouse
>> input. I also see similar behaviour whenever TRAMP is busy with the
>> network, say authenticating to the remote server. As soon as its done,
>> every thing goes back to normal. Maybe I should have taken screen
>> shots. :-\
>
> Please apply
>
>    (setq debug-on-error t
>          debug-on-signal t
>          debug-on-quit t)
>
> before the next test. Then, when Emacs hangs, you could try from a shell
>
> # kill -FPE<emacspid>
>
> Don't ask me why, but sometimes Emacs comes back with the FPE signal,
> and you get a backtrace. This I would like to see, because then I could
> protect that code with-local-quit and friends.
>

These are the steps I took to generate the backtrace, after setting the 
variables as you requested.

* Opened a remote file and edited it.
* Disconnected my network connection.
* Tried to save the file with `C-x C-s'
* After emacs hung, I took a screenshot
* Killed emacs with `kill -FPE <emacs_pid>'
* Copied the backtrace to a buffer, saved it as raw-text.
* Executed the kill command again (as emacs was still unresponsive)

I have attached the screenshot and the text file with the backtrace.

Hope this will help. And as always, thanks for the help.

> Best regards, Michael.

-- 
Suvayu

Open source is the future. It sets us free.

[-- Attachment #2: hung-emacs.png --]
[-- Type: image/png, Size: 20637 bytes --]

[-- Attachment #3: tramp-traceback-2.txt --]
[-- Type: text/plain, Size: 5987 bytes --]

Debugger entered--Lisp error: (arith-error)
  accept-process-output(#<process *tramp/ssh lxplus*> 1 nil)
  byte-code("Æ\b†\x06\0ÇÈÉ	$‰\x12\vB\x13Ê\f\b\r#\x16\vȇ" [timeout with-timeout-tag with-timeout-timer with-timeout-timers proc timeout-msecs run-with-timer 1 nil with-timeout-handler accept-process-output with-timeout-value] 6)
  tramp-accept-process-output(#<process *tramp/ssh lxplus*> 1)
  tramp-wait-for-regexp(#<process *tramp/ssh lxplus*> nil "\r?\n///ff2516d703f6a0738e269a12ea812c6f\r?\n")
  tramp-wait-for-output(#<process *tramp/ssh lxplus*>)
  tramp-send-command(["ssh" nil "lxplus" "/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h"] "test -e /afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h 2>/dev/null; echo tramp_exit_status $? ")
  tramp-send-command-and-check(["ssh" nil "lxplus" "/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h"] "test -e /afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  tramp-handle-file-exists-p("/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  apply(tramp-handle-file-exists-p "/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  tramp-sh-file-name-handler(file-exists-p "/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  apply(tramp-sh-file-name-handler file-exists-p "/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  tramp-file-name-handler(file-exists-p "/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  file-exists-p("/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  tramp-handle-file-attributes("/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  apply(tramp-handle-file-attributes "/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  tramp-sh-file-name-handler(file-attributes "/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  apply(tramp-sh-file-name-handler file-attributes "/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  tramp-file-name-handler(file-attributes "/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  file-attributes("/ssh:lxplus:/afs/cern.ch/user/s/sali/public/athena/15.5.3.11/DataQuality/DataQualityTools/DataQualityTools/DQTMuonIDTrackTool.h")
  tramp-handle-verify-visited-file-modtime(#<buffer DQTMuonIDTrackTool.h>)
  apply(tramp-handle-verify-visited-file-modtime #<buffer DQTMuonIDTrackTool.h>)
  tramp-sh-file-name-handler(verify-visited-file-modtime #<buffer DQTMuonIDTrackTool.h>)
  apply(tramp-sh-file-name-handler verify-visited-file-modtime #<buffer DQTMuonIDTrackTool.h>)
  tramp-file-name-handler(verify-visited-file-modtime #<buffer DQTMuonIDTrackTool.h>)
  verify-visited-file-modtime(#<buffer DQTMuonIDTrackTool.h>)
  basic-save-buffer()
  save-buffer()
  #[(buffer) "r\bqˆÁ )‡" [buffer save-buffer] 1](#<buffer DQTMuonIDTrackTool.h>)
  map-y-or-n-p(#[(buffer) "Å\b!…V\0Æ\b!…V\0Ç\b!?…V\0È\b!„+\0	…V\0\bqˆ\n…V\0É ÊV…V\0Ë	!ƒ;\0r\bqˆ	 )…V\0\vƒA\0̇Ì\x14È\b!ƒP\0ÍÎÈ\b!\"‡ÍÏÐ\b!\"‡" [buffer pred buffer-offer-save arg queried buffer-live-p buffer-modified-p buffer-base-buffer buffer-file-name buffer-size 0 functionp t format "Save file %s? " "Save buffer %s? " buffer-name] 4] #[(buffer) "r\bqˆÁ )‡" [buffer save-buffer] 1] (#<buffer DQTMuonIDTrackTool.h> #<buffer *scratch*> #<buffer  *Minibuf-1*> #<buffer tramp-traceback.txt> #<buffer  *Minibuf-0*> #<buffer *Messages*> #<buffer  *code-conversion-work*> #<buffer  *server*> #<buffer  *Echo Area 0*> #<buffer  *Echo Area 1*> #<buffer *tramp/ssh lxplus*> #<buffer *Completions*>) ("buffer" "buffers" "save") ((18 #[... "\b„\r\0Ã	!ˆ	\x12‚\x15\0Ä	Å\"ˆÆ ˆÇ‡" [enable-recursive-minibuffers buf other-window-scroll-buffer display-buffer view-buffer #[... "À ‡" [exit-recursive-edit] 1] recursive-edit nil] 3] "view this buffer") (100 #[... "Ã\b!„\r\0ÄÅ!ˆ‚)\0Æ‹ˆ	„\x1f\0ÇÈÉ!!ˆÉ\x12‚)\0ÊÈÉ!Ë\"ˆÌ ˆÍ‡" [buf enable-recursive-minibuffers other-window-scroll-buffer buffer-file-name message "Not applicable: no file" ... display-buffer get-buffer-create "*Diff*" view-buffer #[... "À ‡" [exit-recursive-edit] 1] recursive-edit nil] 3] "view changes in this buffer")))
  byte-code("Ɖ\x18\x19Ɖ\x1a^[Ç Æ\x1c‰\x1dƒ/\0\r@\x14r\fqˆ\x0e\x18ƒ'\0È ƒ'\0É\x11Ê ˆ)\rA‰\x15„\x11\0*ËÌÍÇ Î\x0e\x19%\x12\x0e\x1aƒc\0\x0e^[ƒc\0\x0e\x1c„Z\0\x0e\x1aÏ=„Z\0ÐÑÒ\x0e\x1d\"!ƒ^\0ÓÆ!ˆÆ\x16^[É\x13\b†|\0\nÔV†|\0\v†|\0Õ	ƒz\0Ö‚{\0×!,‡" [queried some-automatic files-done abbrevs-done buffer --cl-dolist-temp-- nil buffer-list buffer-modified-p t save-buffer map-y-or-n-p #[(buffer) "Å\b!…V\0Æ\b!…V\0Ç\b!?…V\0È\b!„+\0	…V\0\bqˆ\n…V\0É ÊV…V\0Ë	!ƒ;\0r\bqˆ	 )…V\0\vƒA\0̇Ì\x14È\b!ƒP\0ÍÎÈ\b!\"‡ÍÏÐ\b!\"‡" [buffer pred buffer-offer-save arg queried buffer-live-p buffer-modified-p buffer-base-buffer buffer-file-name buffer-size 0 functionp t format "Save file %s? " "Save buffer %s? " buffer-name] 4] #[(buffer) "r\bqˆÁ )‡" [buffer save-buffer] 1] ("buffer" "buffers" "save") silently y-or-n-p format "Save abbrevs in %s? " write-abbrev-file 0 message "(Some special files were saved without asking)" "(No files need saving)" buffer-save-without-query save-some-buffers-action-alist save-abbrevs abbrevs-changed arg abbrev-file-name] 7)
  save-some-buffers(nil)
  call-interactively(save-some-buffers nil nil)

Re: Killing a hung ssh process in a TRAMP session

[Michael Albinus]

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> Hi Michael,

Hi,

> These are the steps I took to generate the backtrace, after setting
> the variables as you requested.
>
> * Opened a remote file and edited it.
> * Disconnected my network connection.
> * Tried to save the file with `C-x C-s'
> * After emacs hung, I took a screenshot
> * Killed emacs with `kill -FPE <emacs_pid>'
> * Copied the backtrace to a buffer, saved it as raw-text.
> * Executed the kill command again (as emacs was still unresponsive)
>
> I have attached the screenshot and the text file with the backtrace.
>
> Hope this will help. And as always, thanks for the help.

I could reproduce it. Unfortunately, there is no simple way to solve it
inside Tramp (but I will continue to investigate). Therefore, I have
added the following item to Tramp's Frequently Asked Questions, HTH:

   * TRAMP does not recognize hung `ssh' sessions

     When your network connection is down, `ssh' sessions might hang.
     TRAMP cannot detect it safely, because it still sees a running
     `ssh' process.  Timeouts cannot be used as well, because it
     cannot be predicted, how long a remote command will last, for
     example when copying very large files.

     Therefore, you must configure the `ssh' process to die in such a
     case.  The following entry in `~/.ssh/config' would do the job:

          Host *
               ServerAliveInterval 5

Best regards, Michael.

Re: Killing a hung ssh process in a TRAMP session

[Suvayu Ali]

Hi Michael,

On Sunday 17 January 2010 04:23 AM, Michael Albinus wrote:
> Suvayu Ali<fatkasuvayu+linux@gmail.com>  writes:
>> These are the steps I took to generate the backtrace, after setting
>> the variables as you requested.
>
> I could reproduce it. Unfortunately, there is no simple way to solve it
> inside Tramp (but I will continue to investigate). Therefore, I have
> added the following item to Tramp's Frequently Asked Questions, HTH:

Thank you so much for clarifying this issue. :)

>     * TRAMP does not recognize hung `ssh' sessions
>
>       When your network connection is down, `ssh' sessions might hang.
>       TRAMP cannot detect it safely, because it still sees a running
>       `ssh' process.  Timeouts cannot be used as well, because it
>       cannot be predicted, how long a remote command will last, for
>       example when copying very large files.
>
>       Therefore, you must configure the `ssh' process to die in such a
>       case.  The following entry in `~/.ssh/config' would do the job:
>
>            Host *
>                 ServerAliveInterval 5

I was thinking something along similar lines, good to see I was on the 
correct track. :)

> Best regards, Michael.

Cheers,
-- 
Suvayu

Open source is the future. It sets us free.