bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Sebastian Rose]

Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs@gnu.org mailing list,
and to the gnu.emacs.bug news group.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:


If I do

  M-x ediff-files RET

and choose binary files, e.g. x.png and x.jpg, this is, what I see in
the `*ediff*' buffer (no indentation): 

  Bin\303\244rdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind verschieden.


What I expect to see is:

  Binärdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind verschieden.



What I'd prefer even more, would be `ediff' to load those two files two
different buffers, put those buffers in hexl-mode and diff them.

Before `ediff' does this, it should ask (maybe ask only, if the two
files or one of them exceeds a certain size).









If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/usr/local/share/emacs/23.1/etc/DEBUG for instructions.


In GNU Emacs 23.1.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.16.1)
 of 2009-11-16 on beteigeuze
Windowing system distributor `The X.Org Foundation', version 11.0.10600000
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: de_DE.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  recentf-mode: t
  show-paren-mode: t
  global-hl-line-mode: t
  display-time-mode: t
  desktop-save-mode: t
  auto-image-file-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  global-auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
a s SPC s o n t <backspace> s t ? <down> M-> <return> 
<return> <return> SPC SPC S E b ' <backspace> <backspace> 
<backspace> e b a s t i a n <up> <up> <up> <up> <up> 
<right> <return> <return> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <backspace> SPC a <backspace> 
A b e r SPC i c h SPC w e i s SPC n i c h t SPC w e 
<backspace> <backspace> <M-backspace> <M-backspace> 
<M-backspace> <M-backspace> <up> <right> M-c <right> 
<right> <right> <right> SPC g a r <down> <down> <down> 
C-c C-c <right> <left> d E E <switch-frame> q y C-x 
k <return> <S-up> q C-x o C-x 1 C-x k <return> C-x 
C-f <up> <up> <return> C-x C-f <up> <up> <return> M-x 
e i <backspace> d i <tab> f <tab> <tab> - f i <tab> 
<tab> s <return> <up> <up> <return> <up> <up> <return> 
q C-x k C-g C-SPC <down> M-w <switch-frame> <S-left> 
<down> <down> <up> SPC <down> <down> <down> <down> 
<down> <up> <up> <up> <up> <up> E E E E E SPC E E E 
E E E E SPC E E E SPC <up> <up> SPC <down> <down> <down> 
<up> <up> SPC <down> SPC <down> <down> <down> <down> 
<down> <up> E E E E E <up> <up> <up> <up> <up> <up> 
<up> E <switch-frame> <up> C-SPC <down> M-w M-x e <backspace> 
- b u g <tab> M-b <backspace> r e p o r t - <tab> 
<return>

Recent messages:
Quit
Mark set
Retrieving newsgroup: nnml:list:emacs-devel...
Fetching headers for nnml:list:emacs-devel...done
Scoring...done
Generating summary...done
No more unread articles
No more articles
Mark activated
Making completion list...

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Kevin Rodgers]

Sebastian Rose wrote:
> What I'd prefer even more, would be `ediff' to load those two files two
> different buffers, put those buffers in hexl-mode and diff them.

That's what ediff-buffers is for.

> Before `ediff' does this, it should ask (maybe ask only, if the two
> files or one of them exceeds a certain size).

Maybe ediff should have a option whether ediff-files visits the files
and runs ediff-buffers, or retains its original behavior.

-- 
Kevin Rodgers
Denver, Colorado, USA

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Lars Ingebrigtsen]

Sebastian Rose <sebastian_rose@gmx.de> writes:

> If I do
>
>   M-x ediff-files RET
>
> and choose binary files, e.g. x.png and x.jpg, this is, what I see in
> the `*ediff*' buffer (no indentation): 
>
>   Bin\303\244rdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind verschieden.
>
> What I expect to see is:
>
>   Binärdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind verschieden.

I don't have a German locale here, so I can't test this.  However, the
buffer I get when I try this is seems to use an utf-8 input method, so
it seems likely that this has been fixed in the decade that passed since
the bug report.

Does somebody here have Emacs running in a German locale and can test
this?  (Or any other locale that outputs non-ASCII characters for "diff
bin1 bin2".)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Stephen Berman]

On Thu, 20 Aug 2020 20:38:04 +0200 Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Sebastian Rose <sebastian_rose@gmx.de> writes:
>
>> If I do
>>
>>   M-x ediff-files RET
>>
>> and choose binary files, e.g. x.png and x.jpg, this is, what I see in
>> the `*ediff*' buffer (no indentation): 
>>
>>   Bin\303\244rdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind
>> verschieden.
>>
>> What I expect to see is:
>>
>>   Binärdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind verschieden.
>
> I don't have a German locale here, so I can't test this.  However, the
> buffer I get when I try this is seems to use an utf-8 input method, so
> it seems likely that this has been fixed in the decade that passed since
> the bug report.
>
> Does somebody here have Emacs running in a German locale and can test
> this?  (Or any other locale that outputs non-ASCII characters for "diff
> bin1 bin2".)

When I start Emacs built from current master like this:

LANG=de_DE.UTF-8 emacs -Q

and then carry out the above recipe, I get exactly the above message
with raw bytes instead of a-umlaut.

Steve Berman

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Date: Thu, 20 Aug 2020 20:53:12 +0200
> Cc: Sebastian Rose <sebastian_rose@gmx.de>, 5050@debbugs.gnu.org
> 
> >> If I do
> >>
> >>   M-x ediff-files RET
> >>
> >> and choose binary files, e.g. x.png and x.jpg, this is, what I see in
> >> the `*ediff*' buffer (no indentation): 
> >>
> >>   Bin\303\244rdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind
> >> verschieden.
> >>
> >> What I expect to see is:
> >>
> >>   Binärdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind verschieden.
> >
> > I don't have a German locale here, so I can't test this.  However, the
> > buffer I get when I try this is seems to use an utf-8 input method, so
> > it seems likely that this has been fixed in the decade that passed since
> > the bug report.
> >
> > Does somebody here have Emacs running in a German locale and can test
> > this?  (Or any other locale that outputs non-ASCII characters for "diff
> > bin1 bin2".)
> 
> When I start Emacs built from current master like this:
> 
> LANG=de_DE.UTF-8 emacs -Q
> 
> and then carry out the above recipe, I get exactly the above message
> with raw bytes instead of a-umlaut.

What do you see in that environment if you diff 2 files that have text
with umlauts? do you also see raw bytes?

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Lars Ingebrigtsen]

Stephen Berman <stephen.berman@gmx.net> writes:

> When I start Emacs built from current master like this:
>
> LANG=de_DE.UTF-8 emacs -Q
>
> and then carry out the above recipe, I get exactly the above message
> with raw bytes instead of a-umlaut.

Thanks for checking.  I've now installed the locale here, and I'm also
able to reproduce the bug.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Stephen Berman]

On Thu, 20 Aug 2020 22:21:55 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Date: Thu, 20 Aug 2020 20:53:12 +0200
>> Cc: Sebastian Rose <sebastian_rose@gmx.de>, 5050@debbugs.gnu.org
>> 
>> >> If I do
>> >>
>> >>   M-x ediff-files RET
>> >>
>> >> and choose binary files, e.g. x.png and x.jpg, this is, what I see in
>> >> the `*ediff*' buffer (no indentation): 
>> >>
>> >>   Bin\303\244rdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind
>> >> verschieden.
>> >>
>> >> What I expect to see is:
>> >>
>> >>   Binärdateien /home/sebastian/x.png and /home/sebastian/x.jpg sind verschieden.
>> >
>> > I don't have a German locale here, so I can't test this.  However, the
>> > buffer I get when I try this is seems to use an utf-8 input method, so
>> > it seems likely that this has been fixed in the decade that passed since
>> > the bug report.
>> >
>> > Does somebody here have Emacs running in a German locale and can test
>> > this?  (Or any other locale that outputs non-ASCII characters for "diff
>> > bin1 bin2".)
>> 
>> When I start Emacs built from current master like this:
>> 
>> LANG=de_DE.UTF-8 emacs -Q
>> 
>> and then carry out the above recipe, I get exactly the above message
>> with raw bytes instead of a-umlaut.
>
> What do you see in that environment if you diff 2 files that have text
> with umlauts? do you also see raw bytes?

Do you mean with Ediff?  If so, I see the text with umlauts, not with
raw bytes.

Steve Berman

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: larsi@gnus.org,  sebastian_rose@gmx.de,  5050@debbugs.gnu.org
> Date: Thu, 20 Aug 2020 21:35:30 +0200
> 
> > What do you see in that environment if you diff 2 files that have text
> > with umlauts? do you also see raw bytes?
> 
> Do you mean with Ediff?  If so, I see the text with umlauts, not with
> raw bytes.

So we decode the output written to stdout, but not to stderr, is that
the case?

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Lars Ingebrigtsen]

(defun ediff-exec-process (program buffer synch options &rest files)

[...]

	;; If this is a buffer job, we are diffing temporary files
	;; produced by Emacs with ediff-coding-system-for-write, so
	;; use the same encoding to read the results.
	(coding-system-for-read
	 (if (string-match "buffer" (symbol-name ediff-job-name))
	     ediff-coding-system-for-write
	   ediff-coding-system-for-read))
	args)

This is the problem: ediff-coding-system-for-read is raw-text, which
gives us the binary data in the buffer instead of characters.

Hm...  Looks like it's been raw-text since basically forever.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 20 Aug 2020 21:42:47 +0200
> Cc: Sebastian Rose <sebastian_rose@gmx.de>, 5050@debbugs.gnu.org
> 
> (defun ediff-exec-process (program buffer synch options &rest files)
> 
> [...]
> 
> 	;; If this is a buffer job, we are diffing temporary files
> 	;; produced by Emacs with ediff-coding-system-for-write, so
> 	;; use the same encoding to read the results.
> 	(coding-system-for-read
> 	 (if (string-match "buffer" (symbol-name ediff-job-name))
> 	     ediff-coding-system-for-write
> 	   ediff-coding-system-for-read))
> 	args)
> 
> This is the problem: ediff-coding-system-for-read is raw-text, which
> gives us the binary data in the buffer instead of characters.

How else can you compare files which could have different encodings?

And anyway, Stephen just answered my question, and the diffs are
displayed correctly with non-ASCII characters, not as raw bytes.  So
there's more here than meets the eye.

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Stephen Berman]

On Thu, 20 Aug 2020 21:42:47 +0200 Lars Ingebrigtsen <larsi@gnus.org> wrote:

> (defun ediff-exec-process (program buffer synch options &rest files)
>
> [...]
>
> 	;; If this is a buffer job, we are diffing temporary files
> 	;; produced by Emacs with ediff-coding-system-for-write, so
> 	;; use the same encoding to read the results.
> 	(coding-system-for-read
> 	 (if (string-match "buffer" (symbol-name ediff-job-name))
> 	     ediff-coding-system-for-write
> 	   ediff-coding-system-for-read))
> 	args)
>
> This is the problem: ediff-coding-system-for-read is raw-text, which
> gives us the binary data in the buffer instead of characters.

Indeed, and when I set ediff-coding-system-for-read to utf-8 and carry
out the OP's recipe, the German text is displayed as it should be with
umlaut.

Steve Berman

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> And anyway, Stephen just answered my question, and the diffs are
> displayed correctly with non-ASCII characters, not as raw bytes.  So
> there's more here than meets the eye.

They are, but the output in that buffer is just raw bytes -- no matter
whether the output is from stderr or stdout.  There must be a decoding
thing that takes place later, but I'm still looking for where that
happens.  (I'm wholly unfamiliar with the ediff code.)

When I've found that, the fix would be to apply whatever it's doing to
the error buffer, too...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: sebastian_rose@gmx.de,  stephen.berman@gmx.net,  5050@debbugs.gnu.org
> Date: Thu, 20 Aug 2020 22:01:57 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And anyway, Stephen just answered my question, and the diffs are
> > displayed correctly with non-ASCII characters, not as raw bytes.  So
> > there's more here than meets the eye.
> 
> They are, but the output in that buffer is just raw bytes -- no matter
> whether the output is from stderr or stdout.  There must be a decoding
> thing that takes place later, but I'm still looking for where that
> happens.  (I'm wholly unfamiliar with the ediff code.)
> 
> When I've found that, the fix would be to apply whatever it's doing to
> the error buffer, too...

Wait, the problematic text appears in the Ediff error buffer, right?
So we should perhaps simply decode it?  Does evaluating the below in
that buffer fixes the problem?

  M-: (decode-coding-region (point-min) (point-max) 'undecided) RET

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Date: Thu, 20 Aug 2020 21:51:44 +0200
> Cc: Sebastian Rose <sebastian_rose@gmx.de>, 5050@debbugs.gnu.org
> 
> > This is the problem: ediff-coding-system-for-read is raw-text, which
> > gives us the binary data in the buffer instead of characters.
> 
> Indeed, and when I set ediff-coding-system-for-read to utf-8 and carry
> out the OP's recipe, the German text is displayed as it should be with
> umlaut.

Does setting ediff-coding-system-for-read to utf-8 botch comparison of
files that are encoded in something other than UTF-8?  I think it
will, so this is not the right solution.  Our problem is only with the
messages that end up in the Ediff error buffer.

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> They are, but the output in that buffer is just raw bytes -- no matter
> whether the output is from stderr or stdout.  There must be a decoding
> thing that takes place later, but I'm still looking for where that
> happens.  (I'm wholly unfamiliar with the ediff code.)

But it's not doing that -- it's loading the files into normal Emacs
buffers, so the normal Emacs autodetecting of coding systems takes
place.

Hm...  to me it looks like it never decodes the output from diff -- it
doesn't care what's on the lines, it just uses the diff codes.

Here's what the output buffer looks like:


[-- Attachment #2: Type: image/png, Size: 24286 bytes --]

[-- Attachment #3: Type: text/plain, Size: 343 bytes --]


This is a buffer that the user doesn't see -- instead the real two
buffers are displayed, and the fontising is done there.

But again, I'm not familiar with the code here, but it looks to me that
that's what ediff-extract-diffs is doing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Wait, the problematic text appears in the Ediff error buffer, right?
> So we should perhaps simply decode it?  Does evaluating the below in
> that buffer fixes the problem?
>
>   M-: (decode-coding-region (point-min) (point-max) 'undecided) RET

Yes.  So doing that (in the error buffer before displaying it) is
probably the right fix.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Wait, the problematic text appears in the Ediff error buffer, right?
>> So we should perhaps simply decode it?  Does evaluating the below in
>> that buffer fixes the problem?
>>
>>   M-: (decode-coding-region (point-min) (point-max) 'undecided) RET
>
> Yes.  So doing that (in the error buffer before displaying it) is
> probably the right fix.

Yup, I've found the right place to do the decoding, and I've pushed the
fix to Emacs 28.

Thanks everybody for testing and advice. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Stephen Berman]

On Thu, 20 Aug 2020 22:24:53 +0200 Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> Wait, the problematic text appears in the Ediff error buffer, right?
>>> So we should perhaps simply decode it?  Does evaluating the below in
>>> that buffer fixes the problem?
>>>
>>>   M-: (decode-coding-region (point-min) (point-max) 'undecided) RET
>>
>> Yes.  So doing that (in the error buffer before displaying it) is
>> probably the right fix.
>
> Yup, I've found the right place to do the decoding, and I've pushed the
> fix to Emacs 28.
>
> Thanks everybody for testing and advice.

FTR, I confirm that this fix results in the correct display of the
German error message.  So I think the bug can be closed, unless we want
to wait for additional confirmation from the OP.

Steve Berman

bug#5050: 23.1; efiff: Displaying bytes instead of characters in messages

[Lars Ingebrigtsen]

Stephen Berman <stephen.berman@gmx.net> writes:

> FTR, I confirm that this fix results in the correct display of the
> German error message.  So I think the bug can be closed, unless we want
> to wait for additional confirmation from the OP.

I went ahead and closed it -- it seems pretty likely that it's fixed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no