bug#20117: 25.0.50; Tramp tests failing on Cygwin

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Ken Brown]

Several Tramp tests have recently started failing on Cygwin, the first
being tramp-test19-directory-files-and-attributes.  The test aborts
after the failure:

Test tramp-test19-directory-files-and-attributes aborted with non-local exit
   ABORTED  20/35  tramp-test19-directory-files-and-attributes

Aborted: Ran 35 tests, 18 results as expected, 1 unexpected (2015-03-16 
09:36:52-0400)

[...]

Error running tests
   backtrace()
   (progn (message "Error running tests") (backtrace))
   (unwind-protect (progn (message "Error running tests") (backtrace)) 
(kill-emacs 2))
   (unwind-protect (let ((stats (ert-run-tests-batch selector))) 
(kill-emacs (if (= 0 (ert-stats-completed-unexpected stats)) 0 1))) 
(unwind-protect (progn (message "Error running tests") (backtrace)) 
(kill-emacs 2)))
   ert-run-tests-batch-and-exit()
   command-line-1(("-L" ":." "-l" "ert" "-l" "tramp-tests.elc" "-f" 
"ert-run-tests-batch-and-exit"))
   command-line()
   normal-top-level()
Makefile:85: recipe for target 'tramp-tests.log' failed


A bisection shows that the problem started with the following commit:

commit 8ac08792a71e07bce0e62d93bed80553fbe95a40
Author: Michael Albinus <michael.albinus@gmx.de>
Date:   Wed Mar 4 11:02:36 2015 +0100

     Sync with Tramp upstream.

I don't know much about Tramp (or ert), but if someone gives me
instructions, I'll do what I can to help track this down.

Ken

In GNU Emacs 25.0.50.3 (x86_64-unknown-cygwin, GTK+ Version 3.14.9)
  of 2015-03-16 on moufang
Repository revision: b9d09d80d2bf98e5df65316c4e85b448380f7ca7
Windowing system distributor `The Cygwin/X Project', version 11.0.11701000
Configured using:
  `configure --without-all --cache-file=/tmp/config.cache'

Configured features:


Important settings:
   value of $LANG: en_US.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Achim Gratz]

Ken Brown writes:
> Several Tramp tests have recently started failing on Cygwin, the first
> being tramp-test19-directory-files-and-attributes.  The test aborts
> after the failure:

Tramp connections to Cygwin also don't work anymore from newer Emacs and
just hang.  I think that is because Tramp is trying to use ssh control
master which doesn't work on Cygwin.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Michael Albinus]

Ken Brown <kbrown@cornell.edu> writes:

> Several Tramp tests have recently started failing on Cygwin, the first
> being tramp-test19-directory-files-and-attributes.  The test aborts
> after the failure:
>
> Test tramp-test19-directory-files-and-attributes aborted with non-local exit
>   ABORTED  20/35  tramp-test19-directory-files-and-attributes

All tests in tramp-tests.el can be instrumented like this:

--8<---------------cut here---------------start------------->8---
*** /home/albinus/src/emacs/test/automated/tramp-tests.el.~master~	2015-03-16 19:42:03.629467429 +0100
--- /home/albinus/src/emacs/test/automated/tramp-tests.el	2015-03-16 17:41:30.112292199 +0100
***************
*** 1024,1029 ****
--- 1024,1030 ----
    "Check `directory-files-and-attributes'."
    (skip-unless (tramp--test-enabled))
  
+   (tramp--instrument-test-case 10
    ;; `directory-files-and-attributes' contains also values for "../".
    ;; Ensure that this doesn't change during tests, for
    ;; example due to handling temporary files.
***************
*** 1059,1065 ****
  	       (equal (file-attributes (car elt)) (cdr elt)))))
  	  (setq attr (directory-files-and-attributes tmp-name2 nil "^b"))
  	  (should (equal (mapcar 'car attr) '("bar" "boz"))))
!       (ignore-errors (delete-directory tmp-name1 'recursive)))))
  
  (ert-deftest tramp-test20-file-modes ()
    "Check `file-modes'.
--- 1060,1066 ----
  	       (equal (file-attributes (car elt)) (cdr elt)))))
  	  (setq attr (directory-files-and-attributes tmp-name2 nil "^b"))
  	  (should (equal (mapcar 'car attr) '("bar" "boz"))))
!       (ignore-errors (delete-directory tmp-name1 'recursive))))))
  
  (ert-deftest tramp-test20-file-modes ()
    "Check `file-modes'.
--8<---------------cut here---------------end--------------->8---

There will be a lot of traces in the test's *Messages* buffer. If you
run the Tramp tests interactively, go with the cursor to the failed test
and type "m". In batchmode, it shall be dumped.

> Ken

Best regards, Michael.

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Michael Albinus]

Achim Gratz <Stromeko@nexgo.de> writes:

> Tramp connections to Cygwin also don't work anymore from newer Emacs and
> just hang.  I think that is because Tramp is trying to use ssh control
> master which doesn't work on Cygwin.

Could you please run the very recent Emacs master with tramp-verbose set
to 6? I would like to see the debug buffer of Tramp.

> Regards,
> Achim.

Best regards, Michael.

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Achim Gratz]

Michael Albinus writes:
> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> Tramp connections to Cygwin also don't work anymore from newer Emacs and
>> just hang.  I think that is because Tramp is trying to use ssh control
>> master which doesn't work on Cygwin.
>
> Could you please run the very recent Emacs master with tramp-verbose set
> to 6? I would like to see the debug buffer of Tramp.

The changes you're likely interested in are not yet in my build.  The
current master doesn't build however, so that'll have to wait a bit:

Eager macro-expansion failure: (void-function cl-every)
Eager macro-expansion failure: (void-function cl-every)
Symbol's function definition is void: cl-every
Makefile:830: recipe for target 'bootstrap-emacs' failed


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

On 3/16/2015 2:48 PM, Michael Albinus wrote:> All tests in 
tramp-tests.el can be instrumented like this:
[...]
> There will be a lot of traces in the test's *Messages* buffer. If you
> run the Tramp tests interactively, go with the cursor to the failed test
> and type "m". In batchmode, it shall be dumped.

Output attached.

Ken


[-- Attachment #2: mycheck.log.gz --]
[-- Type: application/gzip, Size: 69677 bytes --]

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Ken Brown]

On 3/16/2015 4:57 PM, Ken Brown wrote:
> On 3/16/2015 2:48 PM, Michael Albinus wrote:> All tests in
> tramp-tests.el can be instrumented like this:
> [...]
>> There will be a lot of traces in the test's *Messages* buffer. If you
>> run the Tramp tests interactively, go with the cursor to the failed test
>> and type "m". In batchmode, it shall be dumped.
>
> Output attached.

In case it helps, here's the complete list of Tramp tests that currently 
abort on Cygwin:

tramp-test19-directory-files-and-attributes
tramp-test30-special-characters
tramp-test30-special-characters-with-stat
tramp-test31-utf8
tramp-test31-utf8-with-stat

I've instrumented all of them and can make the traces available to you 
if it would help.  The four new ones come to more than 9MB, after 
tarring and compressing.

Ken

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Achim Gratz]

Michael Albinus writes:
>> Tramp connections to Cygwin also don't work anymore from newer Emacs and
>> just hang.  I think that is because Tramp is trying to use ssh control
>> master which doesn't work on Cygwin.
>
> Could you please run the very recent Emacs master with tramp-verbose set
> to 6? I would like to see the debug buffer of Tramp.

The build today took about twice as long and is using much more memory
than before, but it produces a working Emacs and Tramp can use sshx
connections to my Cygwin box again.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Michael Albinus]

Ken Brown <kbrown@cornell.edu> writes:

> In case it helps, here's the complete list of Tramp tests that
> currently abort on Cygwin:
>
> tramp-test19-directory-files-and-attributes
> tramp-test30-special-characters
> tramp-test30-special-characters-with-stat
> tramp-test31-utf8
> tramp-test31-utf8-with-stat
>
> I've instrumented all of them and can make the traces available to you
> if it would help.  The four new ones come to more than 9MB, after
> tarring and compressing.

Likely, all of them are caused by the same problem. The traces you have
sent to me for tramp-test19-directory-files-and-attributes indicate a
problem with the marker I have introduced in order to handle apostrophs
in file names properly. I could reproduce it locally with bash and zsh;
other bournish shells like ksh and dash work properly.

The following patch let the construct work in all four investigated
shells for me, could you pls check?

--8<---------------cut here---------------start------------->8---
*** /home/albinus/src/tramp/lisp/tramp-sh.el.~master~	2015-03-17 20:12:48.781125579 +0100
--- /home/albinus/src/tramp/lisp/tramp-sh.el	2015-03-17 20:05:35.432048823 +0100
***************
*** 1719,1730 ****
       ;; "-"; this would confuse xargs.  "ls -aQ" might be a solution,
       ;; but it does not work on all remote systems.  Therefore, we
       ;; use \000 as file separator.
!      ;; Apostrophes in the stat output are masked as \037 characters, in
       ;; order to make a proper shell escape of them in file names.
       "cd %s && echo \"(\"; (%s %s -a | "
       "xargs %s -c "
!      "'(\037%%n\037 (\037%%N\037) %%h %s %s %%Xe0 %%Ye0 %%Ze0 %%se0 \037%%A\037 t %%ie0 -1)'"
!      " -- 2>/dev/null | sed -e 's/\"/\\\\\"/g' -e 's/\037/\"/g'); echo \")\"")
      (tramp-shell-quote-argument localname)
      (tramp-get-ls-command vec)
      ;; On systems which have no quoting style, file names with
--- 1719,1730 ----
       ;; "-"; this would confuse xargs.  "ls -aQ" might be a solution,
       ;; but it does not work on all remote systems.  Therefore, we
       ;; use \000 as file separator.
!      ;; Apostrophes in the stat output are masked as \006 characters, in
       ;; order to make a proper shell escape of them in file names.
       "cd %s && echo \"(\"; (%s %s -a | "
       "xargs %s -c "
!      "'(\006%%n\006 (\006%%N\006) %%h %s %s %%Xe0 %%Ye0 %%Ze0 %%se0 \006%%A\006 t %%ie0 -1)'"
!      " -- 2>/dev/null | sed -e 's/\"/\\\\\"/g' -e 's/\006/\"/g'); echo \")\"")
      (tramp-shell-quote-argument localname)
      (tramp-get-ls-command vec)
      ;; On systems which have no quoting style, file names with
***************
*** 1732,1739 ****
      (if (tramp-get-ls-command-with-quoting-style vec)
  	"--quoting-style=shell" "")
      (tramp-get-remote-stat vec)
!     (if (eq id-format 'integer) "%ue0" "\037%U\037")
!     (if (eq id-format 'integer) "%ge0" "\037%G\037"))))
  
  ;; This function should return "foo/" for directories and "bar" for
  ;; files.
--- 1732,1739 ----
      (if (tramp-get-ls-command-with-quoting-style vec)
  	"--quoting-style=shell" "")
      (tramp-get-remote-stat vec)
!     (if (eq id-format 'integer) "%ue0" "\006%U\006")
!     (if (eq id-format 'integer) "%ge0" "\006%G\006"))))
  
  ;; This function should return "foo/" for directories and "bar" for
  ;; files.
--8<---------------cut here---------------end--------------->8---

> Ken

Best regards, Michael.

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Michael Albinus]

Achim Gratz <Stromeko@nexgo.de> writes:

> The build today took about twice as long and is using much more memory
> than before, but it produces a working Emacs and Tramp can use sshx
> connections to my Cygwin box again.

Thanks. So one of the recent patches might have cured it for you. In
case Tramp ceases to work again, it would be nice if you drop a note (or
a bug report).

> Regards,
> Achim.

Best regards, Michael.

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 557 bytes --]

On 3/17/2015 3:16 PM, Michael Albinus wrote:
> Likely, all of them are caused by the same problem. The traces you have
> sent to me for tramp-test19-directory-files-and-attributes indicate a
> problem with the marker I have introduced in order to handle apostrophs
> in file names properly. I could reproduce it locally with bash and zsh;
> other bournish shells like ksh and dash work properly.
>
> The following patch let the construct work in all four investigated
> shells for me, could you pls check?

The test still fails.  New traces attached.

Ken


[-- Attachment #2: test19.log.gz --]
[-- Type: application/gzip, Size: 63721 bytes --]

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Michael Albinus]

Ken Brown <kbrown@cornell.edu> writes:

>> The following patch let the construct work in all four investigated
>> shells for me, could you pls check?
>
> The test still fails.  New traces attached.

Well, the patch was successful in the sense that the command is accepted
by bash now. But bash doesn't seem to like control characters in its
flow. So let's use "/" as apostroph marker; it is very unlikely that it
is part of any file name :-)

Could you pls test the revised patch?

--8<---------------cut here---------------start------------->8---
*** /tmp/ediff2557DrZ	2015-03-18 13:34:09.986438397 +0100
--- /net/ford/albinus/src/tramp/lisp/tramp-sh.el	2015-03-18 13:32:58.096723817 +0100
***************
*** 1719,1730 ****
       ;; "-"; this would confuse xargs.  "ls -aQ" might be a solution,
       ;; but it does not work on all remote systems.  Therefore, we
       ;; use \000 as file separator.
!      ;; Apostrophes in the stat output are masked as \037 characters, in
       ;; order to make a proper shell escape of them in file names.
       "cd %s && echo \"(\"; (%s %s -a | "
       "xargs %s -c "
!      "'(\037%%n\037 (\037%%N\037) %%h %s %s %%Xe0 %%Ye0 %%Ze0 %%se0 \037%%A\037 t %%ie0 -1)'"
!      " -- 2>/dev/null | sed -e 's/\"/\\\\\"/g' -e 's/\037/\"/g'); echo \")\"")
      (tramp-shell-quote-argument localname)
      (tramp-get-ls-command vec)
      ;; On systems which have no quoting style, file names with
--- 1719,1730 ----
       ;; "-"; this would confuse xargs.  "ls -aQ" might be a solution,
       ;; but it does not work on all remote systems.  Therefore, we
       ;; use \000 as file separator.
!      ;; Apostrophes in the stat output are masked as ?/ characters, in
       ;; order to make a proper shell escape of them in file names.
       "cd %s && echo \"(\"; (%s %s -a | "
       "xargs %s -c "
!      "'(/%%n/ (/%%N/) %%h %s %s %%Xe0 %%Ye0 %%Ze0 %%se0 /%%A/ t %%ie0 -1)'"
!      " -- 2>/dev/null | sed -e 's/\"/\\\\\"/g' -e 's/\\//\"/g'); echo \")\"")
      (tramp-shell-quote-argument localname)
      (tramp-get-ls-command vec)
      ;; On systems which have no quoting style, file names with
***************
*** 1732,1739 ****
      (if (tramp-get-ls-command-with-quoting-style vec)
  	"--quoting-style=shell" "")
      (tramp-get-remote-stat vec)
!     (if (eq id-format 'integer) "%ue0" "\037%U\037")
!     (if (eq id-format 'integer) "%ge0" "\037%G\037"))))
  
  ;; This function should return "foo/" for directories and "bar" for
  ;; files.
--- 1732,1739 ----
      (if (tramp-get-ls-command-with-quoting-style vec)
  	"--quoting-style=shell" "")
      (tramp-get-remote-stat vec)
!     (if (eq id-format 'integer) "%ue0" "/%U/")
!     (if (eq id-format 'integer) "%ge0" "/%G/"))))
  
  ;; This function should return "foo/" for directories and "bar" for
  ;; files.
--8<---------------cut here---------------end--------------->8---

> Ken

Best regards, Michael.

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Ken Brown]

On 3/18/2015 8:37 AM, Michael Albinus wrote:
> Well, the patch was successful in the sense that the command is accepted
> by bash now. But bash doesn't seem to like control characters in its
> flow. So let's use "/" as apostroph marker; it is very unlikely that it
> is part of any file name :-)
>
> Could you pls test the revised patch?

That fixes it.  All Tramp tests finish as expected now.

Thanks.

Ken

bug#20117: 25.0.50; Tramp tests failing on Cygwin

[Michael Albinus]

Version: 25.1

Ken Brown <kbrown@cornell.edu> writes:

>> Could you pls test the revised patch?
>
> That fixes it.  All Tramp tests finish as expected now.

Thanks for confirmation. I've pushed the patch to the master branch.

> Ken

Best regards, Michael.