bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

[John Wiegley]

[-- Attachment #1.1: Type: text/plain, Size: 374 bytes --]

I have a friend who is using Emacs 24.5 on Windows, and who is experiencing a
strange behavior: Unicode quotes display properly, but after *invoking* a
'make' command -- even with its output fully redirected to a file -- buffer
decoding begins to fail, causing later quotes to be display as escape codes.

I'm attaching a screenshot from his machine, to show the behavior:


[-- Attachment #1.2: Untitled.png --]
[-- Type: image/png, Size: 743095 bytes --]

[-- Attachment #1.3: Type: text/plain, Size: 452 bytes --]


Note that in a fresh shell, cat'ing the saved file (test-output) does NOT mess
up the quotes. So it's a mysterious side-effect of calling 'make'. Also,
saying 'make xxxx' where xxxx is not a valid target, produces the appropriate
error message and does not mess up the quotes.

Any ideas?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

[Simon Peyton Jones]

Thanks John.

It's emacs 24.5.1 (686-pc-mingw32)

It's not 'make' per se; for example 'make --version' or 'make boot' (which does nothing) is fine.  The invocation of make that causes the funny quotes to appear invokes Python, if that helps.

Deeply mysterious.

If I start a new shell, I get quotes looking nice again.  Could something in the shell environment be affected by invoking python?

Simon

| -----Original Message-----
| From: John Wiegley [mailto:jwiegley@gmail.com]
| Sent: 19 October 2016 18:33
| To: bug-gnu-emacs@gnu.org
| Cc: Simon Peyton Jones <simonpj@microsoft.com>
| Subject: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode
| 
| I have a friend who is using Emacs 24.5 on Windows, and who is
| experiencing a strange behavior: Unicode quotes display properly, but
| after *invoking* a 'make' command -- even with its output fully
| redirected to a file -- buffer decoding begins to fail, causing later
| quotes to be display as escape codes.
| 
| I'm attaching a screenshot from his machine, to show the behavior:

bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

[Eli Zaretskii]

> From: Simon Peyton Jones <simonpj@microsoft.com>
> Date: Wed, 19 Oct 2016 19:07:44 +0000
> 
> It's not 'make' per se; for example 'make --version' or 'make boot' (which does nothing) is fine.  The invocation of make that causes the funny quotes to appear invokes Python, if that helps.
> 
> Deeply mysterious.
> 
> If I start a new shell, I get quotes looking nice again.  Could something in the shell environment be affected by invoking python?

There aren't enough data in the report, only hints.  The image posted
by John indicates a Unixy shell is also involved, so it could be some
issue with the shell itself or its environment (I'm guessing the shell
is either a Cygwin or an MSYS2 Bash, which might mean the Emacs setup
of running them involves setting some coding-systems to UTF-8, for
example).

Then there are 'make' and Python, which are somehow related, but it's
unclear how, due to lack of a complete description.

In general, this looks like UTF-8 output by a subprocess gets decoded
using the Windows codepage.  But it's impossible to tell where exactly
is the problem without knowing the details.

IOW, a reproducible recipe, starting from "emacs -Q", and including
only those customizations that are needed for the reproduction, is
necessary to analyze this problem.  It is most likely a problem either
with user customization of the coding-systems, or with some subtle
incompatibility between Cygwin/MSYS programs and the native Windows
build of Emacs.

bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

[John Wiegley]

Eli makes an excellent point: We should try starting Emacs with the -Q flag
next, and attempt a reproduction following the same steps.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Simon Peyton Jones <simonpj@microsoft.com>,  24741@debbugs.gnu.org
> Date: Thu, 20 Oct 2016 12:19:30 -0700
> 
> Eli makes an excellent point: We should try starting Emacs with the -Q flag
> next, and attempt a reproduction following the same steps.

Actually, posting here the relevant parts of Simon's customizations
(the ones that are related to coding-systems and to running shells)
might also help identify the probable causes.

bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

[Simon Peyton Jones]

Thanks everyone.  It's a bizarre bug!  But it's one that does make my life a lot worse, because my compiler's error messages are littered with quotes, each of which turns into 12 characters of encoding, as you can see.

If you want me to try emacs -Q, just let me know how to do that and what info to collect.  I've never done that before.

Customisations. In my .emacs I have the following, which John Wiegley gave to me in the first place

(setq-default locale-coding-system 'utf-8-unix)
(setq-default buffer-file-coding-system 'utf-8-unix)
(set-default-coding-systems 'utf-8-unix)
(set-terminal-coding-system 'utf-8-unix)
(set-keyboard-coding-system 'utf-8-unix)
(set-selection-coding-system 'utf-8-unix)
(prefer-coding-system 'utf-8-unix)

(defun utf8-shell ()
  (interactive)
  (set-default-coding-systems 'utf-8-unix)
  (shell))

(setq w32-get-true-file-attributes nil)

Simon


|  -----Original Message-----
|  From: Eli Zaretskii [mailto:eliz@gnu.org]
|  Sent: 21 October 2016 07:43
|  To: John Wiegley <jwiegley@gmail.com>
|  Cc: Simon Peyton Jones <simonpj@microsoft.com>; 24741@debbugs.gnu.org
|  Subject: Re: bug#24741: 25.1.50; Buffer encoding corrupted up by
|  'make' in shell-mode
|  
|  > From: John Wiegley <jwiegley@gmail.com>
|  > Cc: Simon Peyton Jones <simonpj@microsoft.com>,
|  24741@debbugs.gnu.org
|  > Date: Thu, 20 Oct 2016 12:19:30 -0700
|  >
|  > Eli makes an excellent point: We should try starting Emacs with the
|  -Q
|  > flag next, and attempt a reproduction following the same steps.
|  
|  Actually, posting here the relevant parts of Simon's customizations
|  (the ones that are related to coding-systems and to running shells)
|  might also help identify the probable causes.

bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

[Eli Zaretskii]

> From: Simon Peyton Jones <simonpj@microsoft.com>
> CC: "24741@debbugs.gnu.org" <24741@debbugs.gnu.org>, Simon Peyton Jones
> 	<simonpj@microsoft.com>
> Date: Fri, 21 Oct 2016 21:20:34 +0000
> 
> If you want me to try emacs -Q, just let me know how to do that and what info to collect.  I've never done that before.

Start Emacs with "emacs -Q", then try reproducing the problem by doing
whatever you do when the problem normally happens.

If "emacs -Q" lacks some optional features and custom settings of
certain variables, load those features and customize those variables
as part of the recipe (i.e. record everything you need to do starting
from "emacs -Q" until you are able to run the commands that produce
the problematic output).

Then post here the full recipe for reproducing the problem, including
everything you have recorded on the way.

> Customisations. In my .emacs I have the following, which John Wiegley gave to me in the first place
> 
> (setq-default locale-coding-system 'utf-8-unix)
> (setq-default buffer-file-coding-system 'utf-8-unix)
> (set-default-coding-systems 'utf-8-unix)
> (set-terminal-coding-system 'utf-8-unix)
> (set-keyboard-coding-system 'utf-8-unix)
> (set-selection-coding-system 'utf-8-unix)
> (prefer-coding-system 'utf-8-unix)
> 
> (defun utf8-shell ()
>   (interactive)
>   (set-default-coding-systems 'utf-8-unix)
>   (shell))

This is already dead wrong on Windows, especially setting the default
encodings to UTF-8.  It cannot possibly work well on Windows.  And you
shouldn't need this.

What happens if you remove these and try again -- does the problem
still appear?

Where did you get your Bash, your make.exe, and your Python?  If they
are from the MSYS2 project, the only customization of coding-systems
you may need is of process-coding-system-alist.  If you do that, make
sure the encoding is still your system codepage, and only the decoding
part is UTF-8.  That's because the encoding is used to encode the
command-line arguments, and you certainly don't want them to be
encoded in UTF-8 on Windows, because this is unsupported.

bug#24741: 25.1.50; Buffer encoding corrupted up by 'make' in shell-mode

[Glenn Morris]

Eli Zaretskii wrote:

> Start Emacs with "emacs -Q", then try reproducing the problem by doing
> whatever you do when the problem normally happens.
>
> If "emacs -Q" lacks some optional features and custom settings of
> certain variables, load those features and customize those variables
> as part of the recipe (i.e. record everything you need to do starting
> from "emacs -Q" until you are able to run the commands that produce
> the problematic output).
>
> Then post here the full recipe for reproducing the problem, including
> everything you have recorded on the way.
[...]
> This is already dead wrong on Windows, especially setting the default
> encodings to UTF-8.  It cannot possibly work well on Windows.  And you
> shouldn't need this.
>
> What happens if you remove these and try again -- does the problem
> still appear?
>
> Where did you get your Bash, your make.exe, and your Python?  If they
> are from the MSYS2 project, the only customization of coding-systems
> you may need is of process-coding-system-alist.  If you do that, make
> sure the encoding is still your system codepage, and only the decoding
> part is UTF-8.  That's because the encoding is used to encode the
> command-line arguments, and you certainly don't want them to be
> encoded in UTF-8 on Windows, because this is unsupported.

No further reponse, closing.
Possibly a bit soon, so feel free to reopen.