FAILED tramp-test34-utf8

FAILED tramp-test34-utf8

[Colin Baxter]

Hello,

I did a 'make check' for emacs-24.5-rc3-fixed-12006-g7d27f5d and got:

Ran 31 tests, 30 results as expected, 1 unexpected (2017-03-28 15:56:05+0100)

1 unexpected results:
   FAILED  tramp-test34-utf8

The relevant part of the backtrace seems to be

Test tramp-test34-utf8 condition:
    (ert-test-failed
     ((should
       (string-equal
	(funcall ... ...)
	(file-remote-p ... ...)))
      :form
      (string-equal
    "/tmp/tramp-test8351vif/\316\223\317\205\317\201\316\257\317\
203\317\204\316\265 \317\204\316\277
    \316\223\316\261\316\273\316\261\316\276\316\257\316\261
    \316\274\316\265 \316\217\317\204\316\277
    \316\243\317\204\316\277\317\200" "/tmp/tramp-test8351vif/
 ---- various unknown characters removed by me prior to send -----
 ")
      :value nil))

Any suggests how I might proceed to correct things?

My system is 

Linux 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686 GNU/Linux.

Thanks, Colin.

Re: FAILED tramp-test34-utf8

[Michael Albinus]

Colin Baxter <m43cap@yandex.com> writes:

> Hello,

Hi Colin,

> I did a 'make check' for emacs-24.5-rc3-fixed-12006-g7d27f5d and got:
>
> Ran 31 tests, 30 results as expected, 1 unexpected (2017-03-28 15:56:05+0100)
>
> 1 unexpected results:
>    FAILED  tramp-test34-utf8
>
> Any suggests how I might proceed to correct things?

Are you sure you have done this with Emacs 24.5? The respective test
there is called tramp-test31-utf8.

> Thanks, Colin.

Best regards, Michael.

Re: FAILED tramp-test34-utf8

[Colin Baxter]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

Dear Michael,

On Tue, Mar 28 2017, Michael Albinus wrote:

> Colin Baxter <m43cap@yandex.com> writes:
>
>> Hello,
>
> Hi Colin,
>
>> I did a 'make check' for emacs-24.5-rc3-fixed-12006-g7d27f5d and got:
>>
>> Ran 31 tests, 30 results as expected, 1 unexpected (2017-03-28 15:56:05+0100)
>>
>> 1 unexpected results:
>>    FAILED  tramp-test34-utf8
>>
>> Any suggests how I might proceed to correct things?
>
> Are you sure you have done this with Emacs 24.5? The respective test
> there is called tramp-test31-utf8.
>
>> Thanks, Colin.
>
> Best regards, Michael.

I have repeated the 'make test' on a newly git-pulled emacs, using a
different machine (same os: 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686
GNU/Linux). I made certain that the first emacs found in PATH was
emacs-26.0.50.

I obtained the same error as before. I've attached the tramp-test.log.

I think a previous reply from me may not have been sent to this List,
for which I apologise.

Best wishes, Colin.


[-- Attachment #2: tramp-tests.log --]
[-- Type: text/plain, Size: 4123 bytes --]

Running 31 tests (2017-03-29 09:06:08+0100)
Remote directory: `/mock::/tmp/'
   passed   1/31  tramp-test00-availability
   passed   2/31  tramp-test01-file-name-syntax
   passed   3/31  tramp-test02-file-name-dissect
   passed   4/31  tramp-test03-file-name-defaults
   passed   5/31  tramp-test04-substitute-in-file-name
   passed   6/31  tramp-test05-expand-file-name
   passed   7/31  tramp-test06-directory-file-name
   passed   8/31  tramp-test07-file-exists-p
   passed   9/31  tramp-test08-file-local-copy
   passed  10/31  tramp-test09-insert-file-contents
   passed  11/31  tramp-test10-write-region
   passed  12/31  tramp-test11-copy-file
   passed  13/31  tramp-test12-rename-file
   passed  14/31  tramp-test13-make-directory
   passed  15/31  tramp-test14-delete-directory
   passed  16/31  tramp-test15-copy-directory
   passed  17/31  tramp-test16-directory-files
   passed  18/31  tramp-test17-insert-directory
   passed  19/31  tramp-test18-file-attributes
   passed  20/31  tramp-test19-directory-files-and-attributes
   passed  21/31  tramp-test20-file-modes
   passed  22/31  tramp-test21-file-links
   passed  23/31  tramp-test22-file-times
   passed  24/31  tramp-test23-visited-file-modtime
   passed  25/31  tramp-test24-file-name-completion
   passed  26/31  tramp-test25-load
   passed  27/31  tramp-test31-make-auto-save-file-name
   passed  28/31  tramp-test32-make-nearby-temp-file
   passed  29/31  tramp-test33-special-characters
Test tramp-test34-utf8 backtrace:
  (if (unwind-protect (setq value-2552 (apply fn-2550 args-2551)) (set
  (let (form-description-2554) (if (unwind-protect (setq value-2552 (a
  (let ((value-2552 (quote ert-form-evaluation-aborted-2553))) (let (f
  (let ((fn-2550 (function string-equal)) (args-2551 (list (funcall (i
  (progn (make-symbolic-link file1 file3) (let ((fn-2540 (function fil
  (condition-case err (progn (make-symbolic-link file1 file3) (let ((f
  (let* ((file1 (expand-file-name elt tmp-name1)) (file2 (expand-file-
  (let ((elt (car --dolist-tail--))) (let* ((file1 (expand-file-name e
  (while --dolist-tail-- (let ((elt (car --dolist-tail--))) (let* ((fi
  (let ((--dolist-tail-- files)) (while --dolist-tail-- (let ((elt (ca
  (progn (make-directory tmp-name1) (make-directory tmp-name2) (let ((
  (unwind-protect (progn (make-directory tmp-name1) (make-directory tm
  (let* ((tramp-test-temporary-file-directory (file-truename tramp-tes
  (let (quoted) (let* ((tramp-test-temporary-file-directory (file-true
  tramp--test-check-files("Γυρίστε το Γαλαξία με Ώτο Στοπ" "أصبح بوسعك
  (let* ((utf8 (if (and (eq system-type (quote darwin)) (memq (quote u
  tramp--test-utf8()
  (closure (t) nil (let ((fn-2745 (function tramp--test-enabled)) (arg
  ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc
  ert-run-test([cl-struct-ert-test tramp-test34-utf8 "Check UTF8 encod
  ert-run-or-rerun-test([cl-struct-ert--stats (not (tag :expensive-tes
  ert-run-tests((not (tag :expensive-test)) #[385 "\306\x02\307\"\203G\0\2
  ert-run-tests-batch((not (tag :expensive-test)))
  ert-run-tests-batch-and-exit((not (tag :expensive-test)))
  eval((ert-run-tests-batch-and-exit (quote (not (tag :expensive-test)
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/net/tramp-tests.el" 
  command-line()
  normal-top-level()
Test tramp-test34-utf8 condition:
    (ert-test-failed
     ((should
       (string-equal
	(funcall ... ...)
	(file-remote-p ... ...)))
      :form
      (string-equal "/tmp/tramp-test711lou/\316\223\317\205\317\201\316\257\317\203\317\204\316\265 \317\204\316\277 \316\223\316\261\316\273\316\261\316\276\316\257\316\261 \316\274\316\265 \316\217\317\204\316\277 \316\243\317\204\316\277\317\200" "/tmp/tramp-test711lou/Γυρίστε το Γαλαξία με Ώτο Στοπ")
      :value nil))
   FAILED  30/31  tramp-test34-utf8
   passed  31/31  tramp-test36-recursive-load

Ran 31 tests, 30 results as expected, 1 unexpected (2017-03-29 09:06:20+0100)

1 unexpected results:
   FAILED  tramp-test34-utf8

Re: FAILED tramp-test34-utf8

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 620 bytes --]

Colin Baxter <m43cap@yandex.com> writes:

> Dear Michael,

Hi Colin,

> I have repeated the 'make test' on a newly git-pulled emacs, using a
> different machine (same os: 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686
> GNU/Linux). I made certain that the first emacs found in PATH was
> emacs-26.0.50.
>
> I obtained the same error as before. I've attached the tramp-test.log.

Thanks. However, it doesn't tell too much to us.

Pls apply the appended patch to tramp-tests.el. It instruments
`tramp-test34-utf8', writing lot of traces into tramp-tests.log.

Rerun the test, and send tramp-test.log.

> Best wishes, Colin.


[-- Attachment #2: *vc-diff* --]
[-- Type: text/x-patch, Size: 953 bytes --]

diff --git a/test/lisp/net/tramp-tests.el b/test/lisp/net/tramp-tests.el
index 2a4ef74..383d323 100644
--- a/test/lisp/net/tramp-tests.el
+++ b/test/lisp/net/tramp-tests.el
@@ -2463,12 +2463,13 @@ tramp--test-utf8
 	 (coding-system-for-read utf8)
 	 (coding-system-for-write utf8)
 	 (file-name-coding-system utf8))
+    (tramp--instrument-test-case 10
     (tramp--test-check-files
      (unless (tramp--test-hpux-p) "Γυρίστε το Γαλαξία με Ώτο Στοπ")
      (unless (tramp--test-hpux-p)
        "أصبح بوسعك الآن تنزيل نسخة كاملة من موسوعة ويكيبيديا العربية لتصفحها بلا اتصال بالإنترنت")
      "银河系漫游指南系列"
-     "Автостопом по гала́ктике")))
+     "Автостопом по гала́ктике"))))

 (ert-deftest tramp-test34-utf8 ()
   "Check UTF8 encoding in file names and file contents."

Re: FAILED tramp-test34-utf8

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Colin,

> Thanks. However, it doesn't tell too much to us.

PS: I suppose you haven't set your locale to be utf8-aware. What happens,
if you call in a shell

# locale

Best regards, Michael.

Re: FAILED tramp-test34-utf8

[Colin Baxter]

On Wed, Mar 29 2017, Michael Albinus wrote:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
> Hi Colin,
>
>> Thanks. However, it doesn't tell too much to us.
>
> PS: I suppose you haven't set your locale to be utf8-aware. What happens,
> if you call in a shell
>
> # locale
>
This is my screen output:

LANG=en_GB.utf8
LANGUAGE=
LC_CTYPE="en_GB.utf8"
LC_NUMERIC="en_GB.utf8"
LC_TIME="en_GB.utf8"
LC_COLLATE="en_GB.utf8"
LC_MONETARY="en_GB.utf8"
LC_MESSAGES="en_GB.utf8"
LC_PAPER="en_GB.utf8"
LC_NAME="en_GB.utf8"
LC_ADDRESS="en_GB.utf8"
LC_TELEPHONE="en_GB.utf8"
LC_MEASUREMENT="en_GB.utf8"
LC_IDENTIFICATION="en_GB.utf8"
LC_ALL=


Best wishes, Colin.

Re: FAILED tramp-test34-utf8

[Colin Baxter]

On Wed, Mar 29 2017, Michael Albinus wrote:

> Colin Baxter <m43cap@yandex.com> writes:
>
>> Dear Michael,
>
> Hi Colin,
>
>> I have repeated the 'make test' on a newly git-pulled emacs, using a
>> different machine (same os: 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686
>> GNU/Linux). I made certain that the first emacs found in PATH was
>> emacs-26.0.50.
>>
>> I obtained the same error as before. I've attached the tramp-test.log.
>
> Thanks. However, it doesn't tell too much to us.
>
> Pls apply the appended patch to tramp-tests.el. It instruments
> `tramp-test34-utf8', writing lot of traces into tramp-tests.log.
>
> Rerun the test, and send tramp-test.log.
>
>> Best wishes, Colin.

Give me a couple of days to read up on how to apply patches before I
send you the results.

Best wishes, Colin.

Re: FAILED tramp-test34-utf8

[Michael Albinus]

Colin Baxter <m43cap@yandex.com> writes:

Hi Colin,

> This is my screen output:
>
> LANG=en_GB.utf8
> LANGUAGE=
> LC_CTYPE="en_GB.utf8"
> LC_NUMERIC="en_GB.utf8"
> LC_TIME="en_GB.utf8"
> LC_COLLATE="en_GB.utf8"
> LC_MONETARY="en_GB.utf8"
> LC_MESSAGES="en_GB.utf8"
> LC_PAPER="en_GB.utf8"
> LC_NAME="en_GB.utf8"
> LC_ADDRESS="en_GB.utf8"
> LC_TELEPHONE="en_GB.utf8"
> LC_MEASUREMENT="en_GB.utf8"
> LC_IDENTIFICATION="en_GB.utf8"
> LC_ALL=

LGTM. Let's see, what we'll find in the log-file.

> Best wishes, Colin.

Best regards, Michael.

Re: FAILED tramp-test34-utf8

[Eli Zaretskii]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 794 bytes --]

> From: Colin Baxter <m43cap@yandex.com>
> Date: Wed, 29 Mar 2017 09:38:41 +0100
> Cc: , emacs-devel@gnu.org
> 
> Test tramp-test34-utf8 condition:
>     (ert-test-failed
>      ((should
>        (string-equal
> 	(funcall ... ...)
> 	(file-remote-p ... ...)))
>       :form
>       (string-equal "/tmp/tramp-test711lou/\316\223\317\205\317\201\316\257\317\203\317\204\316\265 \317\204\316\277 \316\223\316\261\316\273\316\261\316\276\316\257\316\261 \316\274\316\265 \316\217\317\204\316\277 \316\243\317\204\316\277\317\200" "/tmp/tramp-test711lou/Γυρίστε το Γαλαξία με Ώτο Στοπ")
>       :value nil))

Looks like an attempt to compare a multibyte string with a unibyte
string holding the same bytes.  In such cases, string-equal returns
nil, per its documentation.

Re: FAILED tramp-test34-utf8

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> Looks like an attempt to compare a multibyte string with a unibyte
> string holding the same bytes.  In such cases, string-equal returns
> nil, per its documentation.

You mean it is sheer luck that the test passes usually? Hmm.

Best regards, Michael.

Re: FAILED tramp-test34-utf8

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Colin Baxter <m43cap@yandex.com>,  emacs-devel@gnu.org
> Date: Wed, 29 Mar 2017 19:00:35 +0200
> 
> > Looks like an attempt to compare a multibyte string with a unibyte
> > string holding the same bytes.  In such cases, string-equal returns
> > nil, per its documentation.
> 
> You mean it is sheer luck that the test passes usually? Hmm.

If indeed one of the strings is unibyte, the other multibyte, then
yes.  It's hard to say whether this is the case by looking at the
bytes presented by Colin, since at least some of them seem to have
been converted to ASCII strings \nnn instead of the original bytes.
So maybe my guess about the reason is wrong.

Re: FAILED tramp-test34-utf8

[Colin Baxter]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

Dear Michael,

On Wed, Mar 29 2017, Michael Albinus wrote:

> Colin Baxter <m43cap@yandex.com> writes:
>
>> Dear Michael,
>
> Hi Colin,
>
>> I have repeated the 'make test' on a newly git-pulled emacs, using a
>> different machine (same os: 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686
>> GNU/Linux). I made certain that the first emacs found in PATH was
>> emacs-26.0.50.
>>
>> I obtained the same error as before. I've attached the tramp-test.log.
>
> Thanks. However, it doesn't tell too much to us.
>
> Pls apply the appended patch to tramp-tests.el. It instruments
> `tramp-test34-utf8', writing lot of traces into tramp-tests.log.
>
> Rerun the test, and send tramp-test.log.
>

Ok, I've applied your patch and afterwards ran
make clean
make
make check

Here is the tramp-tests.log as an attachment. I haven't yet learnt how
to attached a file in a gnus mail buffer as a folded line. I hope it
helps.

Best wishes, Colin.


[-- Attachment #2: tramp-tests.log --]
[-- Type: application/octet-stream, Size: 206367 bytes --]

Re: FAILED tramp-test34-utf8

[Eli Zaretskii]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

> From: Colin Baxter <m43cap@yandex.com>
> Date: Wed, 29 Mar 2017 20:40:25 +0100
> Cc: , emacs-devel@gnu.org
> 
> Test tramp-test34-utf8 condition:
>     (ert-test-failed
>      ((should
>        (string-equal
> 	(funcall ... ...)
> 	(file-remote-p ... ...)))
>       :form
>       (string-equal "/tmp/tramp-test5548VYB/\316\223\317\205\317\201\316\257\317\203\317\204\316\265 \317\204\316\277 \316\223\316\261\316\273\316\261\316\276\316\257\316\261 \316\274\316\265 \316\217\317\204\316\277 \316\243\317\204\316\277\317\200" "/tmp/tramp-test5548VYB/Γυρίστε το Γαλαξία με Ώτο Στοπ")

It seems the failure comes from this snippet:

		      (should
		       (string-equal
			(funcall
			 (if quoted 'tramp-compat-file-name-quote 'identity)
			 (car (file-attributes file3)))
			(file-remote-p (file-truename file1) 'localname)))

And it's strange that the first file name in the failed condition has
got all of its bytes displayed as ASCII representation, while the
second one still has the raw bytes.

Another question is why do we see raw bytes at all here?  These should
not be unibyte strings.

Re: FAILED tramp-test34-utf8

[Colin Baxter]

On Wed, Mar 29 2017, Colin Baxter wrote:

> Dear Michael,
>
> On Wed, Mar 29 2017, Michael Albinus wrote:
>
>> Colin Baxter <m43cap@yandex.com> writes:
>>
>>> Dear Michael,
>>
>> Hi Colin,
>>
>>> I have repeated the 'make test' on a newly git-pulled emacs, using a
>>> different machine (same os: 3.2.0-4-686-pae #1 SMP Debian 3.2.86-1 i686
>>> GNU/Linux). I made certain that the first emacs found in PATH was
>>> emacs-26.0.50.
>>>
>>> I obtained the same error as before. I've attached the tramp-test.log.
>>
>> Thanks. However, it doesn't tell too much to us.
>>
>> Pls apply the appended patch to tramp-tests.el. It instruments
>> `tramp-test34-utf8', writing lot of traces into tramp-tests.log.
>>
>> Rerun the test, and send tramp-test.log.
>>
>
> Ok, I've applied your patch and afterwards ran
> make clean
> make
> make check
>
> Here is the tramp-tests.log as an attachment. I haven't yet learnt how
> to attached a file in a gnus mail buffer as a folded line. I hope it
> helps.
>
> Best wishes, Colin.


I should have added that 'network-stream-tests' failed to finish because
port 44331 was already in use by gnutls-serv. This may not be
significant.

Best wishes, Colin.

Re: FAILED tramp-test34-utf8

[Michael Albinus]

Colin Baxter <m43cap@yandex.com> writes:

> Dear Michael,

Hi Colin,

> Here is the tramp-tests.log as an attachment. I haven't yet learnt how
> to attached a file in a gnus mail buffer as a folded line. I hope it
> helps.

Thanks for this. An attachment is perfect, because the file contains
non-ascii chars.

And indeed, I found the problem:

> 20:12:06.761824 tramp-send-command (6) # locale -a
> 20:12:06.795357 tramp-wait-for-regexp (6) # 
> C
> C.UTF-8
> POSIX
> en_GB.utf8

Tramp checks, which locales do exist on your remote machine. This are
just C, C.UTF-8, POSIX and en_GB.utf8. Not so many.

> 20:12:06.795610 tramp-set-connection-property (7) # locale LC_ALL=C

Internally, Tramp keeps a list of acceptable locales. This list contains
en_US.utf8, C.utf8 and en_US.UTF-8. None of them matches the available
locales, Tramp decides to use locale C as last resort. And this let the
utf8 specific test fail.

I've added C.UTF-8 to the list of acceptable locales. This shall be
sufficient, at least in your case (and in many other cases as
well). Committed to the master branch; please pull it and check again.

> Best wishes, Colin.

Best regards, Michael.

Re: FAILED tramp-test34-utf8

[Colin Baxter]

Dear Michael,

On Thu, Mar 30 2017, Michael Albinus wrote:

> Colin Baxter <m43cap@yandex.com> writes:
>
>> Dear Michael,
>
> Hi Colin,
>
>> Here is the tramp-tests.log as an attachment. I haven't yet learnt how
>> to attached a file in a gnus mail buffer as a folded line. I hope it
>> helps.
>
> Thanks for this. An attachment is perfect, because the file contains
> non-ascii chars.
>
> And indeed, I found the problem:
>
>> 20:12:06.761824 tramp-send-command (6) # locale -a
>> 20:12:06.795357 tramp-wait-for-regexp (6) # 
>> C
>> C.UTF-8
>> POSIX
>> en_GB.utf8
>
> Tramp checks, which locales do exist on your remote machine. This are
> just C, C.UTF-8, POSIX and en_GB.utf8. Not so many.
>
>> 20:12:06.795610 tramp-set-connection-property (7) # locale LC_ALL=C
>
> Internally, Tramp keeps a list of acceptable locales. This list contains
> en_US.utf8, C.utf8 and en_US.UTF-8. None of them matches the available
> locales, Tramp decides to use locale C as last resort. And this let the
> utf8 specific test fail.
>
> I've added C.UTF-8 to the list of acceptable locales. This shall be
> sufficient, at least in your case (and in many other cases as
> well). Committed to the master branch; please pull it and check again.
>

Success. I re-pulled emacs and ran 'make check' again. According to the
tramp-tests.log, all 31 tests passed as expected. And thank you for
the interesting information about locales.

Best wishes, Colin.