bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Gemini Lasswell]

When I try to resize an image using an Emacs built without
ImageMagick, either with emacs -Q or my full config, the apostrophe in
the error message is not displayed correctly. To reproduce:

C-x C-f path/to/image-file.jpg RET
+

Result: The error "Can^Yt rescale images without ImageMagick support"
appears in the echo area.

When I define this little command and run it:

(defun my-command ()
  (interactive)
  (error "Can't do this"))

then the typographically correct apostrophe appears in the echo area.

In *Messages*, the first error message appears as:

Can\342\200\231t rescale images without ImageMagick support

although now in report-emacs-bug's message composition buffer, I see
the typograpically correct apostrophe.

In GNU Emacs 26.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.21)
 of 2018-02-09 built on localhost
Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Type C-c C-c or C-c C-x to view the image as text or hex.
image--get-imagemagick-and-warn: Can’t rescale images without ImageMagick support
Configured using:
 'configure
 --prefix=/nix/store/y06nnna2nzr4fx1pbigs67hbjm396ijn-emacs-26.0
 --with-modules --with-x-toolkit=gtk3 --with-xft'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS NOTIFY LIBSELINUX
GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 MODULES
THREADS

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Image[jpeg]

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils map seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib image-mode
easymenu elec-pair time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray minibuffer cl-preloaded
nadvice loaddefs button faces cus-face macroexp files text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 97112 7073)
 (symbols 48 20569 1)
 (miscs 40 47 79)
 (strings 32 28786 1005)
 (string-bytes 1 777748)
 (vectors 16 15028)
 (vector-slots 8 504094 7678)
 (floats 8 58 59)
 (intervals 56 209 0)
 (buffers 992 12))

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Glenn Morris]

Gemini Lasswell wrote:

> When I try to resize an image using an Emacs built without
> ImageMagick, either with emacs -Q or my full config, the apostrophe in
> the error message is not displayed correctly. To reproduce:
>
> C-x C-f path/to/image-file.jpg RET
> +
>
> Result: The error "Can^Yt rescale images without ImageMagick support"
> appears in the echo area.

Present since 25.1. Perhaps a minimal example is:
emacs -Q
(set-buffer-multibyte nil)
(message "can't")

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Glenn Morris]

Glenn Morris wrote:

> Present since 25.1. Perhaps a minimal example is:
> emacs -Q
> (set-buffer-multibyte nil)
> (message "can't")

Although I might be confusing the return value with the displayed message.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Glenn Morris]

Maybe a better example is:

(defun foo ()
  (interactive)
  (error "can't"))
(set-buffer-multibyte nil)
M-x foo

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Fri, 09 Feb 2018 19:29:03 -0500
> Cc: Paul Eggert <eggert@cs.ucla.edu>, 30405@debbugs.gnu.org
> 
> Maybe a better example is:
> 
> (defun foo ()
>   (interactive)
>   (error "can't"))
> (set-buffer-multibyte nil)
> M-x foo

I applied (the obvious) band-aid to image.el, so it no longer shows a
garbled error message.

The more general issue should be fixed on master, as it's too late to
make such changes on the release branch.  Note also that
substitute-command-keys is affected as well, as can be seen by
evaluating the following:

  (progn
    (set-buffer-multibyte nil)
    (substitute-command-keys "can't"))

Basically, anything that produces non-ASCII characters and then shows
that in the echo area while the current buffer is unibyte will hit
this problem.  While Lisp programs that produce literal strings can be
told to take care of that when they use unibyte buffers, the cases
discussed in this bug report happen because we convert ASCII strings
to non-ASCII strings under the hood, so the Lisp programs cannot be
held accountable.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Glenn Morris]

Glenn Morris wrote:

> Maybe a better example is:
>
> (defun foo ()
>   (interactive)
>   (error "can't"))
> (set-buffer-multibyte nil)
> M-x foo

BTW, replace "error" with "message" and the issue does not appear.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Sat, 10 Feb 2018 18:34:02 -0500
> Cc: Paul Eggert <eggert@cs.ucla.edu>, 30405@debbugs.gnu.org
> 
> Glenn Morris wrote:
> 
> > Maybe a better example is:
> >
> > (defun foo ()
> >   (interactive)
> >   (error "can't"))
> > (set-buffer-multibyte nil)
> > M-x foo
> 
> BTW, replace "error" with "message" and the issue does not appear.

Because 'message' doesn't change the quotes.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> Date: Sun, 11 Feb 2018 05:38:50 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: gazally@runbox.com, eggert@cs.ucla.edu, 30405@debbugs.gnu.org
> 
> > BTW, replace "error" with "message" and the issue does not appear.
> 
> Because 'message' doesn't change the quotes.

Oops, ignore me.  That's not the reason.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

On February 11, 2018 5:41:19 AM GMT+02:00, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Sun, 11 Feb 2018 05:38:50 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: gazally@runbox.com, eggert@cs.ucla.edu, 30405@debbugs.gnu.org
> > 
> > > BTW, replace "error" with "message" and the issue does not appear.
> > 
> > Because 'message' doesn't change the quotes.
> 
> Oops, ignore me.  That's not the reason.

The real reason is that 'message' has a string to display, and so can set up
the echo-area buffer according to multibyteness of that string (it does that
using set_message_1), whereas 'princ' and friends cannot do that.  And
'error' calls 'princ'.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Noam Postavsky]

Glenn Morris <rgm@gnu.org> writes:

> Present since 25.1. Perhaps a minimal example is:
> emacs -Q
> (set-buffer-multibyte nil)
> (message "can't")

The issue with message producing fancy quotes is new in 25.1, although
stepping with the debugger, it looks like the root cause is that the
" *Echo Area 0*" buffer becomes unibyte along with the main buffer.  So
the following shows the problem in earlier versions as well:

(set-buffer-multibyte nil)
(message "can\u2019t")

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Fri, 09 Feb 2018 22:07:06 -0500
> Cc: Gemini Lasswell <gazally@runbox.com>, Paul Eggert <eggert@cs.ucla.edu>,
> 	30405@debbugs.gnu.org
> 
> The issue with message producing fancy quotes is new in 25.1, although
> stepping with the debugger, it looks like the root cause is that the
> " *Echo Area 0*" buffer becomes unibyte along with the main buffer.  So
> the following shows the problem in earlier versions as well:
> 
> (set-buffer-multibyte nil)
> (message "can\u2019t")

This is the intended behavior, not a bug.  We make the echo area
buffer unibyte when the message is generated with the current buffer
being unibyte.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Andreas Schwab]

On Feb 10 2018, Eli Zaretskii <eliz@gnu.org> wrote:

> This is the intended behavior, not a bug.  We make the echo area
> buffer unibyte when the message is generated with the current buffer
> being unibyte.

Do we?  What I see in set_message_1 is that we set the echo area buffer
multibyteness to the same as the string to be displayed.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Noam Postavsky <npostavs@users.sourceforge.net>,  eggert@cs.ucla.edu,  30405@debbugs.gnu.org,  gazally@runbox.com
> Date: Sat, 10 Feb 2018 15:15:46 +0100
> 
> What I see in set_message_1 is that we set the echo area buffer
> multibyteness to the same as the string to be displayed.

That function is not called in the use case discussed here (I
initially also thought it was part of the story, but GDB convinced me
otherwise).  The function which is relevant here is
setup_echo_area_for_printing, it is called from PRINTPREPARE.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Paul Eggert]

Eli Zaretskii wrote:
> We make the echo area
> buffer unibyte when the message is generated with the current buffer
> being unibyte.

This made sense back in the 1990s when unibyte was commonly used for text. 
Nowadays, though, wouldn't it make more sense to keep the echo area multibyte? 
The echo area is intended for text, not for binary data.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> Cc: rgm@gnu.org, gazally@runbox.com, 30405@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 10 Feb 2018 10:57:28 -0800
> 
> Eli Zaretskii wrote:
> > We make the echo area
> > buffer unibyte when the message is generated with the current buffer
> > being unibyte.
> 
> This made sense back in the 1990s when unibyte was commonly used for text. 
> Nowadays, though, wouldn't it make more sense to keep the echo area multibyte? 
> The echo area is intended for text, not for binary data.

I don't see how the date outside could matter here.  If you understand
the reason behind the code in question, please describe it, and we can
then discuss whether that reason is still valid in the current
codebase.

I have a guess for why we did that: it's because in Emacs 21 we
displayed raw bytes as Latin-N characters, so non-ASCII text in
unibyte strings needed a unibyte buffer to display it as expected.
But that feature is no longer available, as raw bytes are always
displayed as octal escapes.

The question that bothers me is can a unibyte string inserted or
printed into a multibyte buffer be converted to something that will
display as a non-ASCII character, not as an octal escape.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Paul Eggert]

Eli Zaretskii wrote:
>> Cc: rgm@gnu.org, gazally@runbox.com, 30405@debbugs.gnu.org
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Sat, 10 Feb 2018 10:57:28 -0800
>>
>> Eli Zaretskii wrote:
>>> We make the echo area
>>> buffer unibyte when the message is generated with the current buffer
>>> being unibyte.
>>
>> This made sense back in the 1990s when unibyte was commonly used for text.
>> Nowadays, though, wouldn't it make more sense to keep the echo area multibyte?
>> The echo area is intended for text, not for binary data.
> 
> I don't see how the date outside could matter here.

What I was trying to say is that back in the 1990s it was relatively common for 
people to run Emacs mostly in unibyte mode and to edit files in a Latin-1 
locale, so it was natural for programmers to expect the echo area to be 
consistent with the file being edited. Nowadays we live in a mostly-multibyte 
world, where unibyte inside Emacs is intended only for binary data, and so it's 
no longer a reasonable design choice to have the echo area (which is intended 
for text messages to the user) to be unibyte (which is now intended for binary 
data).

> I have a guess for why we did that: it's because in Emacs 21 we
> displayed raw bytes as Latin-N characters, so non-ASCII text in
> unibyte strings needed a unibyte buffer to display it as expected.
> But that feature is no longer available, as raw bytes are always
> displayed as octal escapes.

Sounds plausible.

> The question that bothers me is can a unibyte string inserted or
> printed into a multibyte buffer be converted to something that will
> display as a non-ASCII character, not as an octal escape.

Surely we can arrange for the latter.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> Cc: npostavs@users.sourceforge.net, rgm@gnu.org, gazally@runbox.com,
>  30405@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 11 Feb 2018 09:26:41 -0800
> 
> > The question that bothers me is can a unibyte string inserted or
> > printed into a multibyte buffer be converted to something that will
> > display as a non-ASCII character, not as an octal escape.
> 
> Surely we can arrange for the latter.

I think we already do.  At least I couldn't find a way to display a
raw byte as a non-ASCII Latin character in a unibyte buffer.  If no
one can, we could probably remove that unibyte/multibyte magic in echo
area.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> Date: Sun, 11 Feb 2018 20:04:19 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: npostavs@users.sourceforge.net, 30405@debbugs.gnu.org, gazally@runbox.com
> 
> > Cc: npostavs@users.sourceforge.net, rgm@gnu.org, gazally@runbox.com,
> >  30405@debbugs.gnu.org
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > Date: Sun, 11 Feb 2018 09:26:41 -0800
> > 
> > > The question that bothers me is can a unibyte string inserted or
> > > printed into a multibyte buffer be converted to something that will
> > > display as a non-ASCII character, not as an octal escape.
> > 
> > Surely we can arrange for the latter.
> 
> I think we already do.  At least I couldn't find a way to display a
> raw byte as a non-ASCII Latin character in a unibyte buffer.  If no
> one can, we could probably remove that unibyte/multibyte magic in echo
> area.

Actually, I can:

  emacs -Q
  M-x set-variable RET unibyte-display-via-language-environment RET t RET
  M-: (set-buffer-multibyte nil) RET
  C-q 0242 SPC

This should display ¢.

So I think we can get rid of making echo-area buffers unibyte, as long
as we make sure that variable is nil (which it is by default).

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

Eli Zaretskii wrote:
>    emacs -Q
>    M-x set-variable RET unibyte-display-via-language-environment RET t RET
>    M-: (set-buffer-multibyte nil) RET
>    C-q 0242 SPC
> 
> This should display ¢.
> 
> So I think we can get rid of making echo-area buffers unibyte, as long
> as we make sure that variable is nil (which it is by default).

Getting rid of it sounds good, but why do we need to worry about 
unibyte-display-via-language-environment? For me, the attached patch does 
display that test as ¢, and it fixes the other test cases reported so far in 
Bug#30405. And yet this patch works without worrying about 
unibyte-display-via-language-environment, even if I run Emacs in a unibyte 
locale like en_US.iso885915 (a practice that's no longer common).

For what it's worth, I'm testing on Ubuntu 16.04 and on Fedora 27, built 
--without-imagicmagick so that I can reproduce the original problem.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Echo-area-multibyteness-is-independent-of-buffer.patch --]
[-- Type: text/x-patch; name="0001-Echo-area-multibyteness-is-independent-of-buffer.patch", Size: 5433 bytes --]

From 1ba9c7c8a0fe8c2ec6866a1dfb6b8c340d7d2139 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 11 Feb 2018 12:26:43 -0800
Subject: [PATCH] Echo area multibyteness is independent of buffer

This fixes a problem where ImageMagick-related diagnostics
were mangled (Bug#30405).
* src/print.c (PRINTPREPARE, printchar, strout):
* src/xdisp.c (setup_echo_area_for_printing, set_message_1):
Don't mess with the multibyteness of the echo buffer.
* src/xdisp.c (message_enable_multibyte): Remove.
All uses removed.
---
 src/lisp.h  |  2 +-
 src/print.c | 17 +++++------------
 src/xdisp.c | 22 ++--------------------
 3 files changed, 8 insertions(+), 33 deletions(-)

diff --git a/src/lisp.h b/src/lisp.h
index a7f0a1d..2c5aaa0 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3613,7 +3613,7 @@ extern Lisp_Object echo_area_buffer[2];
 extern void add_to_log (char const *, ...);
 extern void vadd_to_log (char const *, va_list);
 extern void check_message_stack (void);
-extern void setup_echo_area_for_printing (bool);
+extern void setup_echo_area_for_printing (void);
 extern bool push_message (void);
 extern void pop_message_unwind (void);
 extern Lisp_Object restore_message_unwind (Lisp_Object);
diff --git a/src/print.c b/src/print.c
index b3c0f6f..fbf51aa 100644
--- a/src/print.c
+++ b/src/print.c
@@ -98,8 +98,6 @@ bool print_output_debug_flag EXTERNALLY_VISIBLE = 1;
    ptrdiff_t old_point_byte = -1, start_point_byte = -1;		\
    ptrdiff_t specpdl_count = SPECPDL_INDEX ();				\
    bool free_print_buffer = 0;						\
-   bool multibyte							\
-     = !NILP (BVAR (current_buffer, enable_multibyte_characters));	\
    Lisp_Object original = printcharfun;					\
    if (NILP (printcharfun)) printcharfun = Qt;				\
    if (BUFFERP (printcharfun))						\
@@ -154,7 +152,7 @@ bool print_output_debug_flag EXTERNALLY_VISIBLE = 1;
        print_buffer_pos_byte = 0;					\
      }									\
    if (EQ (printcharfun, Qt) && ! noninteractive)			\
-     setup_echo_area_for_printing (multibyte);
+     setup_echo_area_for_printing ();
 
 #define PRINTFINISH							\
    if (NILP (printcharfun))						\
@@ -303,12 +301,9 @@ printchar (unsigned int ch, Lisp_Object fun)
 	}
       else
 	{
-	  bool multibyte_p
-	    = !NILP (BVAR (current_buffer, enable_multibyte_characters));
-
-	  setup_echo_area_for_printing (multibyte_p);
+	  setup_echo_area_for_printing ();
 	  insert_char (ch);
-	  message_dolog ((char *) str, len, 0, multibyte_p);
+	  message_dolog ((char *) str, len, 0, true);
 	}
     }
 }
@@ -360,11 +355,9 @@ strout (const char *ptr, ptrdiff_t size, ptrdiff_t size_byte,
 	 here, that's the reason we don't call printchar to do the
 	 job.  */
       int i;
-      bool multibyte_p
-	= !NILP (BVAR (current_buffer, enable_multibyte_characters));
 
-      setup_echo_area_for_printing (multibyte_p);
-      message_dolog (ptr, size_byte, 0, multibyte_p);
+      setup_echo_area_for_printing ();
+      message_dolog (ptr, size_byte, 0, true);
 
       if (size == size_byte)
 	{
diff --git a/src/xdisp.c b/src/xdisp.c
index 7c90d32..236a4ae 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -445,11 +445,6 @@ Lisp_Object echo_area_window;
 
 static Lisp_Object Vmessage_stack;
 
-/* True means multibyte characters were enabled when the echo area
-   message was specified.  */
-
-static bool message_enable_multibyte;
-
 /* At each redisplay cycle, we should refresh everything there is to refresh.
    To do that efficiently, we use many optimizations that try to make sure we
    don't waste too much time updating things that haven't changed.
@@ -10947,11 +10942,10 @@ unwind_with_echo_area_buffer (Lisp_Object vector)
 }
 
 
-/* Set up the echo area for use by print functions.  MULTIBYTE_P
-   means we will print multibyte.  */
+/* Set up the echo area for use by print functions.  */
 
 void
-setup_echo_area_for_printing (bool multibyte_p)
+setup_echo_area_for_printing (void)
 {
   /* If we can't find an echo area any more, exit.  */
   if (! FRAME_LIVE_P (XFRAME (selected_frame)))
@@ -10982,11 +10976,6 @@ setup_echo_area_for_printing (bool multibyte_p)
 	}
       TEMP_SET_PT_BOTH (BEG, BEG_BYTE);
 
-      /* Set up the buffer for the multibyteness we need.  */
-      if (multibyte_p
-	  != !NILP (BVAR (current_buffer, enable_multibyte_characters)))
-	Fset_buffer_multibyte (multibyte_p ? Qt : Qnil);
-
       /* Raise the frame containing the echo area.  */
       if (minibuffer_auto_raise)
 	{
@@ -11409,8 +11398,6 @@ set_message (Lisp_Object string)
 {
   eassert (STRINGP (string));
 
-  message_enable_multibyte = STRING_MULTIBYTE (string);
-
   with_echo_area_buffer (0, -1, set_message_1, 0, string);
   message_buf_print = false;
   help_echo_showing_p = false;
@@ -11431,11 +11418,6 @@ set_message_1 (ptrdiff_t a1, Lisp_Object string)
 {
   eassert (STRINGP (string));
 
-  /* Change multibyteness of the echo buffer appropriately.  */
-  if (message_enable_multibyte
-      != !NILP (BVAR (current_buffer, enable_multibyte_characters)))
-    Fset_buffer_multibyte (message_enable_multibyte ? Qt : Qnil);
-
   bset_truncate_lines (current_buffer, message_truncate_lines ? Qt : Qnil);
   if (!NILP (BVAR (current_buffer, bidi_display_reordering)))
     bset_bidi_paragraph_direction (current_buffer, Qleft_to_right);
-- 
2.7.4

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> Cc: npostavs@users.sourceforge.net, 30405@debbugs.gnu.org, gazally@runbox.com
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 11 Feb 2018 12:36:47 -0800
> 
> >    emacs -Q
> >    M-x set-variable RET unibyte-display-via-language-environment RET t RET
> >    M-: (set-buffer-multibyte nil) RET
> >    C-q 0242 SPC
> > 
> > This should display ¢.
> > 
> > So I think we can get rid of making echo-area buffers unibyte, as long
> > as we make sure that variable is nil (which it is by default).
> 
> Getting rid of it sounds good, but why do we need to worry about 
> unibyte-display-via-language-environment?

Because with your patch the following doesn't work as it did before:

  emacs -Q
  M-x set-variable RET unibyte-display-via-language-environment RET t RET
  M-: (set-buffer-multibyte nil) RET
  (defun foo ()
    (interactive)
    (message "cannot ¢")) C-j
  M-x foo RET

(To insert ¢, type "C-q 0242".)  And the same if you replace 'message'
with 'error'.  With your patch, I see an octal escape, not ¢, i.e. the
effect of unibyte-display-via-language-environment is lost.

> And yet this patch works without worrying about 
> unibyte-display-via-language-environment, even if I run Emacs in a unibyte 
> locale like en_US.iso885915 (a practice that's no longer common).

Whether the locale is unibyte or not is immaterial, because we use
multibyte representation for Latin-N characters as well.  What is
important is to be in a unibyte buffer and display a message with
non-ASCII bytes, while unibyte-display-via-language-environment is
non-nil.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Paul Eggert]

On 02/12/2018 10:21 AM, Eli Zaretskii wrote:
>> Getting rid of it sounds good, but why do we need to worry about 
>> unibyte-display-via-language-environment? 
> Because with your patch the following doesn't work as it did before: 
> emacs -Q M-x set-variable RET unibyte-display-via-language-environment 
> RET t RET M-: (set-buffer-multibyte nil) RET (defun foo () 
> (interactive) (message "cannot ¢")) C-j M-x foo RET (To insert ¢, type 
> "C-q 0242".) And the same if you replace 'message' with 'error'. With 
> your patch, I see an octal escape, not ¢, i.e. the effect of 
> unibyte-display-via-language-environment is lost. 

That's OK, as the effect is intended for unibyte buffers and strings 
which are intended for binary data, whereas the echo area is intended to 
be text. That is, you're correct that there is a change in behavior 
here; the change is an improvement.

The example that you gave is more clearly formulated as follows:

   emacs -Q
   M-x set-variable RET unibyte-display-via-language-environment RET t RET
   (defun foo ()
     (interactive)
     (message "cannot \242")) C-j
   M-x foo RET

because the message deliberately contains the binary byte with octal 
value 242. It's more appropriate for this example to display "\242" in 
the echo area than to display "¢", because the echo area is text, not 
binary data.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> Cc: npostavs@users.sourceforge.net, 30405@debbugs.gnu.org, gazally@runbox.com
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 12 Feb 2018 11:34:13 -0800
> 
>    emacs -Q
>    M-x set-variable RET unibyte-display-via-language-environment RET t RET
>    (defun foo ()
>      (interactive)
>      (message "cannot \242")) C-j
>    M-x foo RET
> 
> because the message deliberately contains the binary byte with octal 
> value 242. It's more appropriate for this example to display "\242" in 
> the echo area than to display "¢", because the echo area is text, not 
> binary data.

I disagree, because unibyte-display-via-language-environment
explicitly requests display of raw bytes as Latin-1 characters, and it
requests that everywhere, including the echo area and whatnot.  That's
the whole raison d'être of that feature.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Paul Eggert]

On 02/12/2018 11:59 AM, Eli Zaretskii wrote:
> unibyte-display-via-language-environment
> explicitly requests display of raw bytes as Latin-1 characters, and it
> requests that everywhere, including the echo area and whatnot.

That's not how Emacs works, at least not in my experience. For example, 
on current emacs-26:

    emacs -Q
    M-x set-variable RET unibyte-display-via-language-environment RET t RET
    (defun foo ()
      (interactive)
      (message "cannot \xA2\u00A2")) C-j
    M-x foo RET

This displays "\242¢", not "¢¢".

No doubt this isn't documented as well as it should be, but from looking 
at the source code get_next_display_element it's clear that 
unibyte-display-via-language-environment does not simply display every 
raw byte as a Latin-1 character; instead, the code also takes context 
into account, and if the context is multibyte then 
unibyte-display-via-language-environment is ignored. Since the echo 
area's context is text and not binary data, the display of raw bytes in 
the echo area should be unaffected by 
unibyte-display-via-language-environment.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> Cc: npostavs@users.sourceforge.net, 30405@debbugs.gnu.org, gazally@runbox.com
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 12 Feb 2018 12:31:55 -0800
> 
> On 02/12/2018 11:59 AM, Eli Zaretskii wrote:
> > unibyte-display-via-language-environment
> > explicitly requests display of raw bytes as Latin-1 characters, and it
> > requests that everywhere, including the echo area and whatnot.
> 
> That's not how Emacs works, at least not in my experience. For example, 
> on current emacs-26:
> 
>     emacs -Q
>     M-x set-variable RET unibyte-display-via-language-environment RET t RET
>     (defun foo ()
>       (interactive)
>       (message "cannot \xA2\u00A2")) C-j
>     M-x foo RET
> 
> This displays "\242¢", not "¢¢".

Because you shoot your self in the foot by passing a multibyte string
to 'message'.  In this scenario, you are not supposed to do that, you
are supposed to use only unibyte strings.

Also, did you try the variant with 'error' instead of 'message' (in
which case you need to make *scratch* unibyte before invoking 'foo'.

> Since the echo area's context is text and not binary data, the
> display of raw bytes in the echo area should be unaffected by
> unibyte-display-via-language-environment.

That variable's purpose is to display raw bytes as readable text, so I
definitely disagree in this specific use case.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Paul Eggert]

On 02/12/2018 09:03 PM, Eli Zaretskii wrote:
> In this scenario, you are not supposed to do that, you
> are supposed to use only unibyte strings.

In that case the user should set the echo area to be unibyte, and if 
there's not a convenient way to do that then we should supply one. In 
the meantime Emacs is messed up, since it just guesses whether the echo 
area should be unibyte and (as we've seen) it guesses wrong in common cases.

> Also, did you try the variant with 'error' instead of 'message' (in
> which case you need to make*scratch*  unibyte before invoking 'foo'.

In that setup in the emacs-26 branch, (error "\xA2\u00A2") displays "¢¢" 
in the echo area and "\242¢" in *Backtrace* and "\300\242\302\242" in 
*Messages*, which is bogus. The 'message' variant displays "\242¢" in 
all three places; this is much better behavior.

>> Since the echo area's context is text and not binary data, the
>> display of raw bytes in the echo area should be unaffected by
>> unibyte-display-via-language-environment.
> That variable's purpose is to display raw bytes as readable text, so I
> definitely disagree in this specific use case.

The abovementioned test case establishes that the variable does not in 
fact always cause Emacs to display raw bytes as readable text. The only 
question is whether the documentation is wrong, or the code (or both 
:-). I've given a consistent interpretation that the intent of the 
variable is to display raw bytes as text when in a unibyte context 
(which the echo area is not). I haven't seen an alternative consistent 
interpretation that's corresponds to the behavior Emacs currently 
exhibits (i.e., the sort of behavior that elicited this bug report).

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> Cc: npostavs@users.sourceforge.net, 30405@debbugs.gnu.org, gazally@runbox.com
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 13 Feb 2018 09:43:34 -0800
> 
> > That variable's purpose is to display raw bytes as readable text, so I
> > definitely disagree in this specific use case.
> 
> The abovementioned test case establishes that the variable does not in 
> fact always cause Emacs to display raw bytes as readable text. The only 
> question is whether the documentation is wrong, or the code (or both 
> :-). I've given a consistent interpretation that the intent of the 
> variable is to display raw bytes as text when in a unibyte context 
> (which the echo area is not). I haven't seen an alternative consistent 
> interpretation that's corresponds to the behavior Emacs currently 
> exhibits (i.e., the sort of behavior that elicited this bug report).

That's your POV, but not mine.  I'm not prepared to lose this
variable, given the dozen line it takes to support it.  Fixing the
original problem without losing the effect of this variable is easy,
so I see no reason to continue arguing.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Eli Zaretskii]

> Date: Tue, 13 Feb 2018 21:58:30 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: gazally@runbox.com, 30405@debbugs.gnu.org, npostavs@users.sourceforge.net
> 
> > Cc: npostavs@users.sourceforge.net, 30405@debbugs.gnu.org, gazally@runbox.com
> 
> Fixing the original problem without losing the effect of this
> variable is easy

Fixed.

I'm marking this bug done now.

bug#30405: 26.0.91; Incorrect apostrophe translation in ImageMagick error message

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This made sense back in the 1990s when unibyte was commonly used for text. 
  > Nowadays, though, wouldn't it make more sense to keep the echo area multibyte? 
  > The echo area is intended for text, not for binary data.

I agree in principle.  But I don't know how much work the change would
be.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.