bug#12148: 24.1.50; Tramp has problems with Solaris server (test -e)

bug#12148: 24.1.50; Tramp has problems with Solaris server (test -e)

[Carsten Bormann]

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from `emacs -Q':

-- find a file via tramp on a Solaris 5.11 server (e.g., .bash_history)
-- exit emacs, so that a .emacs.d/tramp is written
-- find the file again

-> Error Couldn't find exit status of `test -e /users/home/cabo/.bash_history'

Solaris /bin/sh does not support test -e.
Worse, there is strange behavior when you use it:

bash$ /bin/sh
$ test -e foo ; echo $?
test: argument expected
$

So anything after the failing test does not get executed, which appears to destroy tramp's
exit status management.

This appears to get circumvented correctly once, but the .emacs.d/tramp
written out from the first emacs says:

  ("file-exists" "test -e")

which is clearly wrong.
So the next incarnation of emacs will no longer work correcly with that
Solaris server.

Workaround: kill emacs, edit ~/.emacs.d/tramp to say

  ("file-exists" "/bin/test -e")

and restart emacs.
(Because of the caching in .emacs.d/tramp and the behavior of writing out the tramp file during kill-emacs, this can be very confusing to debug.  It took me a while to understand that the "works/doesn't work/works/doesn't work" behavior can only be explained by to the tramp file written out with bad information.)

In GNU Emacs 24.1.50.1 (x86_64-apple-darwin, NS apple-appkit-1038.36)
 of 2012-08-04 on bob.porkrind.org
Bzr revision: 109423 eggert@cs.ucla.edu-20120804005210-ydbx97z1govlzuik
Windowing system distributor `Apple', version 10.3.1187
Configured using:
 `configure '--host=x86_64-apple-darwin' '--build=i686-apple-darwin'
 '--with-ns' 'build_alias=i686-apple-darwin'
 'host_alias=x86_64-apple-darwin' 'CC=gcc -mmacosx-version-min=10.5''

Important settings:
  value of $LC_CTYPE: UTF-8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Dired by name

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
C-x C-f / a : <return> <return> <help-echo> <menu-bar> 
<help-menu> <send-emacs-bug-report>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Tramp: Opening connection for a using scpc...
Tramp: Sending command `exec ssh   -o ControlPath=/var/folders/hy/4kd7dc593jn5wbqln6_hj_940000gn/T/tramp.28882-39.%r@%h:%p -o ControlMaster=yes -e none a'

Tramp: Waiting for prompts from remote shell
Tramp: Sending command `exec ssh   -o ControlPath=/var/folders/hy/4kd7dc593jn5wbqln6_hj_940000gn/T/tramp.28882-39.%r@%h:%p -o ControlMaster=yes -e none a'
Tramp: Found remote shell prompt on `a'
Tramp: Opening connection for a using scpc...done
ls does not support --dired; see `dired-use-ls-dired' for more details.
byte-code: Couldn't find exit status of `test -e /users/home/cabo/.bash_history'

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rfc822 mml easymenu mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mail-utils dired
tramp-cache tramp-sh tramp tramp-compat auth-source eieio byte-opt
bytecomp byte-compile cconv macroexp gnus-util mm-util mail-prsvr
password-cache tramp-loaddefs shell pcomplete comint ansi-color ring
format-spec advice help-fns advice-preload time-date tooltip ediff-hook
vc-hooks lisp-float-type mwheel ns-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment lisp-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer loaddefs
button faces cus-face files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process ns multi-tty emacs)

bug#12148: 24.1.50; Tramp has problems with Solaris server (test -e)

[Michael Albinus]

Carsten Bormann <cabo@tzi.org> writes:

> This appears to get circumvented correctly once, but the .emacs.d/tramp
> written out from the first emacs says:
>
>   ("file-exists" "test -e")
>
> which is clearly wrong.
> So the next incarnation of emacs will no longer work correcly with that
> Solaris server.

Hmm. In `tramp-find-file-exists-command', it is correctly tested for
"test -e", "/bin/test -e" aso. But the cache seems to get the wrong value.

I would need to analyze the cache operations. Would you, please, apply
"M-x tramp-cleanup-all-connections" (this also removes cache settings),
set `tramp-verbose' to 9, and rerun the test? The resulting debug buffer
shall tell us more.

> (Because of the caching in .emacs.d/tramp and the behavior of writing out the tramp file during kill-emacs, this can be very confusing to debug.  It took me a while to understand that the "works/doesn't work/works/doesn't work" behavior can only be explained by to the tramp file written out with bad information.)

Maybe the Tramp info file shall be improved for recipes how to disable
the cache for debugging.

Best regards, Michael.

bug#12148: 24.1.50; Tramp has problems with Solaris server (test -e)

[Carsten Bormann]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]


On Aug 7, 2012, at 10:51, Michael Albinus <michael.albinus@gmx.de> wrote:

> Carsten Bormann <cabo@tzi.org> writes:
> 
>> This appears to get circumvented correctly once, but the .emacs.d/tramp
>> written out from the first emacs says:
>> 
>>  ("file-exists" "test -e")
>> 
>> which is clearly wrong.
>> So the next incarnation of emacs will no longer work correcly with that
>> Solaris server.
> 
> Hmm. In `tramp-find-file-exists-command', it is correctly tested for
> "test -e", "/bin/test -e" aso. But the cache seems to get the wrong value.
> 
> I would need to analyze the cache operations. Would you, please, apply
> "M-x tramp-cleanup-all-connections" (this also removes cache settings),
> set `tramp-verbose' to 9, and rerun the test? The resulting debug buffer
> shall tell us more.

Interesting.  With tramp-cleanup-all-connections I didn't need the two-step process to trigger the bug.
Again, the problem is that test -e suppresses the rest of the command line in /bin/sh on Solaris, so the first "test -e" fails right away, but is somehow recovered using another attempt that uses test -d. The attempt to find-file (open) .bash_history then completely fails on what seems to be the same kind of "test -e" (hmm, why didn't it do that in my previous tests before I wrote the tramp file).

resulting tramp file as well as debug buffer (slightly sanitized) attached.

Grüße, Carsten


[-- Attachment #2: tramp --]
[-- Type: application/octet-stream, Size: 665 bytes --]

;; -*- emacs-lisp -*- <12/08/06 17:23:35 /Users/cabo/.emacs.d/tramp>
;; Tramp connection history.  Don't change this file.
;; You can delete it, forcing Tramp to reapply the checks.

((["scpc" nil "a" nil nil]
  ("uname" "SunOS 5.11")
  ("test" "test")
  ("remote-path"
   ("/usr/xpg4/bin" "/usr/ccs/bin" "/usr/bin" "/bin" "/usr/sbin" "/usr/local/bin"))
  ("busybox" nil)
  ("remote-shell" "/usr/bin/bash")
  ("~" "/users/home/cabo")
  ("file-exists" "test -e")
  ("perl-file-spec" t)
  ("perl-cwd-realpath" t)
  ("perl" "\\perl")
  ("ls" "/usr/xpg4/bin/ls")
  ("ls-dired" nil)
  ("stat" nil)
  ("id" "/usr/xpg4/bin/id")
  ("gid-integer" 1003)
  ("readlink" nil)))

[-- Attachment #3: debug-tramp-scpc-a --]
[-- Type: application/octet-stream, Size: 68770 bytes --]

bug#12148: 24.1.50; Tramp has problems with Solaris server (test -e)

[Michael Albinus]

Carsten Bormann <cabo@tzi.org> writes:

Hi Carsten,

> Again, the problem is that test -e suppresses the rest of the command
> line in /bin/sh on Solaris, so the first "test -e" fails right away,
> but is somehow recovered using another attempt that uses test -d. The
> attempt to find-file (open) .bash_history then completely fails on
> what seems to be the same kind of "test -e" (hmm, why didn't it do
> that in my previous tests before I wrote the tramp file).

I've hardened the test a little bit. Could you, please, check whether
this works (clean the cache first):

--8<---------------cut here---------------start------------->8---
*** ~/src/tramp/lisp/tramp-sh.el.~2.77~	2012-08-08 08:40:59.692792871 +0200
--- ~/src/tramp/lisp/tramp-sh.el	2012-08-08 08:40:33.696337454 +0200
***************
*** 3559,3584 ****
      ;;                    `/usr/bin/test'.
      ;; `/usr/bin/test -e' In case `/bin/test' does not exist.
      (unless (or
!              (and (setq result (format "%s -e" (tramp-get-test-command vec)))
! 		  (tramp-send-command-and-check
! 		   vec (format "%s %s" result existing))
!                   (not (tramp-send-command-and-check
! 			vec (format "%s %s" result nonexistent))))
!              (and (setq result "/bin/test -e")
! 		  (tramp-send-command-and-check
! 		   vec (format "%s %s" result existing))
!                   (not (tramp-send-command-and-check
! 			vec (format "%s %s" result nonexistent))))
!              (and (setq result "/usr/bin/test -e")
! 		  (tramp-send-command-and-check
! 		   vec (format "%s %s" result existing))
!                   (not (tramp-send-command-and-check
! 			vec (format "%s %s" result nonexistent))))
!              (and (setq result (format "%s -d" (tramp-get-ls-command vec)))
! 		  (tramp-send-command-and-check
! 		   vec (format "%s %s" result existing))
!                   (not (tramp-send-command-and-check
! 			vec (format "%s %s" result nonexistent)))))
        (tramp-error
         vec 'file-error "Couldn't find command to check if file exists"))
      result))
--- 3559,3588 ----
      ;;                    `/usr/bin/test'.
      ;; `/usr/bin/test -e' In case `/bin/test' does not exist.
      (unless (or
! 	     (ignore-errors
! 	       (and (setq result (format "%s -e" (tramp-get-test-command vec)))
! 		    (tramp-send-command-and-check
! 		     vec (format "%s %s" result existing))
! 		    (not (tramp-send-command-and-check
! 			  vec (format "%s %s" result nonexistent)))))
! 	     (ignore-errors
! 	       (and (setq result "/bin/test -e")
! 		    (tramp-send-command-and-check
! 		     vec (format "%s %s" result existing))
! 		    (not (tramp-send-command-and-check
! 			  vec (format "%s %s" result nonexistent)))))
! 	     (ignore-errors
! 	       (and (setq result "/usr/bin/test -e")
! 		    (tramp-send-command-and-check
! 		     vec (format "%s %s" result existing))
! 		    (not (tramp-send-command-and-check
! 			  vec (format "%s %s" result nonexistent)))))
! 	     (ignore-errors
! 	       (and (setq result (format "%s -d" (tramp-get-ls-command vec)))
! 		    (tramp-send-command-and-check
! 		     vec (format "%s %s" result existing))
! 		    (not (tramp-send-command-and-check
! 			  vec (format "%s %s" result nonexistent))))))
        (tramp-error
         vec 'file-error "Couldn't find command to check if file exists"))
      result))
--8<---------------cut here---------------end--------------->8---

> Grüße, Carsten

Best regards, Michael.

bug#12148: 24.1.50; Tramp has problems with Solaris server (test -e)

[Carsten Bormann]

On Aug 8, 2012, at 08:47, Michael Albinus <michael.albinus@gmx.de> wrote:

> I've hardened the test a little bit. Could you, please, check whether
> this works (clean the cache first):

Thanks.

I hope I did this cache-cleaning right by just calling tramp-cleanup-all-connections.

Now, the first find-file (to a directory) leads to an empty buffer (but only in 2 out of 3 attempts?  No idea what I did different.) or just hangs.
Doing that again (after a quit in case of the hang) appears to work properly, as does editing a file.
When I first tried this, an equivalent to  ("file-exists" "test -e")  was not recorded at all, see below.
On another attempt it does get recorded (see second ~/.emacs.d/tramp below).
In the latter case, we then run into the same problem (byte-code: Couldn't find exit status of `test -e /users/home/cabo/.bash_history') with the second incarnation of emacs.

I'm afraid this one didn't quite work out.
Also I'd love to know why this appears to be less than deterministic.

Grüße, Carsten



;; -*- emacs-lisp -*- <12/08/08 09:14:43 /Users/cabo/.emacs.d/tramp>
;; Tramp connection history.  Don't change this file.
;; You can delete it, forcing Tramp to reapply the checks.

((["scpc" nil "a" nil nil]
  ("uname" "SunOS 5.11")
  ("test" "test")
  ("remote-path"
   ("/usr/xpg4/bin" "/usr/ccs/bin" "/usr/bin" "/bin" "/usr/sbin" "/usr/local/bin"))
  ("busybox" nil)
  ("remote-shell" "/usr/bin/bash")
  ("stat" nil)
  ("perl-file-spec" t)
  ("perl-cwd-realpath" t)
  ("perl" "\\perl")
  ("id" "/usr/xpg4/bin/id")
  ("gid-integer" 1003)))

;; -*- emacs-lisp -*- <12/08/08 09:26:01 /Users/cabo/.emacs.d/tramp>
;; Tramp connection history.  Don't change this file.
;; You can delete it, forcing Tramp to reapply the checks.

((["scpc" nil "a" nil nil]
  ("uname" "SunOS 5.11")
  ("test" "test")
  ("remote-path"
   ("/usr/xpg4/bin" "/usr/ccs/bin" "/usr/bin" "/bin" "/usr/sbin" "/usr/local/bin"))
  ("busybox" nil)
  ("remote-shell" "/usr/bin/bash")
  ("~" "/users/home/cabo")
  ("ls" "/usr/xpg4/bin/ls")
  ("ls-dired" nil)
  ("stat" nil)
  ("perl-file-spec" t)
  ("perl-cwd-realpath" t)
  ("perl" "\\perl")
  ("id" "/usr/xpg4/bin/id")
  ("gid-integer" 1003)
  ("file-exists" "test -e")
  ("readlink" nil)))

bug#12148: 24.1.50; Tramp has problems with Solaris server (test -e)

[Michael Albinus]

Carsten Bormann <cabo@tzi.org> writes:

Hi Carsten,

> I hope I did this cache-cleaning right by just calling tramp-cleanup-all-connections.

It should do the job. However, in order to be on the safe side, please
stop Emacs, remove ~/.emacs.d/tramp, and restart Emacs.

> I'm afraid this one didn't quite work out.
> Also I'd love to know why this appears to be less than deterministic.

Strange. During my local tests of this patch, I did obfuscate the first
check in `tramp-find-file-exists-command', in order to provoke a syntax
error. It works fine here.

Please rerun the test from the beginning. And I need the debug buffer,
also from the beginning of the test, with `tramp-verbose' set to 9.

> Grüße, Carsten

Best regards, Michael.

bug#12148: 24.1.50; Tramp has problems with Solaris server (test -e)

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> Please rerun the test from the beginning. And I need the debug buffer,
> also from the beginning of the test, with `tramp-verbose' set to 9.

A modified version of the published patch solves the problem, as
confirmed by Carsten off-list. Patch committed, closing the bug.

Best regards, Michael.