bug#24637: 25.1; non-ascii chars fail to render in Info buffer

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[ivan]

[-- Attachment #1: Type: text/plain, Size: 4109 bytes --]

While reading the Emacs manual in Emacs using Info-mode, I found that
non-ascii characters are displayed as octal escape sequences rather than
rendered as glyphs. To repro:

- start emacs with -Q
- open the emacs manual and start a search: C-h r s
- search for a section with non-ascii chars: it requotes text typed<return>
- the found sentence reads:
> For example, it requotes text typed `like this' to text \342\200\230like
this\342\200\231.
- if correctly rendered, it would read:
> For example, it requotes text typed `like this' to text ‘like this’.

If I select the text and write the region to a file, I can open that file
and Emacs renders it correctly. I tried to remedy this by adding the
following to my init file (and commenting out the rest of the file):

(prefer-coding-system 'utf-8)
(set-default-coding-systems 'utf-8)
(set-terminal-coding-system 'utf-8)
(set-keyboard-coding-system 'utf-8)
(setq default-buffer-file-coding-system 'utf-8)

The only effect this had was to change the default coding system for new
files from utf-8-unix to utf-8. Running `describe-coding` in the Info
buffer reported
> Coding system for saving this buffer: t -- raw-text-unix
both before and after adding the above settings.

I also tried both gui and terminal, with and without the above settings,
and saw the octal escape sequences in all cases.

My Emacs version and configuration:

In GNU Emacs 25.1.1 (x86_64-apple-darwin15.6.0, Carbon Version 157 AppKit
1404.47)
 of 2016-09-21 built on MachineCode.local
Windowing system distributor 'Apple Inc.', version 10.11.6
Configured using:
 'configure --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs-mac/emacs-25.1-z-mac-6.0/share/info/emacs
 --prefix=/usr/local/Cellar/emacs-mac/emacs-25.1-z-mac-6.0 --with-mac
 --enable-mac-app=/usr/local/Cellar/emacs-mac/emacs-25.1-z-mac-6.0'

Configured features:
NOTIFY ACL LIBXML2 ZLIB TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Info

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mac-mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a. [2 times]

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode cl-loaddefs pcase
cl-lib mail-prsvr mail-utils jka-compr info easymenu mule-util time-date
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel mac-win term/common-win tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote kqueue mac multi-tty
make-network-process emacs)

Memory information:
((conses 16 113689 6270)
 (symbols 48 20181 0)
 (miscs 40 73 185)
 (strings 32 19396 4598)
 (string-bytes 1 510009)
 (vectors 16 12204)
 (vector-slots 8 426977 6597)
 (floats 8 169 118)
 (intervals 56 4622 16)
 (buffers 976 19))

[-- Attachment #2: Type: text/html, Size: 5203 bytes --]

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Eli Zaretskii]

> From: ivan <ivan.brennan@gmail.com>
> Date: Fri, 7 Oct 2016 08:13:37 -0400
> 
> - start emacs with -Q
> - open the emacs manual and start a search: C-h r s
> - search for a section with non-ascii chars: it requotes text typed<return>
> - the found sentence reads:
> > For example, it requotes text typed `like this' to text \342\200\230like this\342\200\231.
> - if correctly rendered, it would read:
> > For example, it requotes text typed `like this' to text ‘like this’.

I cannot reproduce this, so this might be something specific to OS X
(unlikely) or to your system.

The emacs.info file supplied with Emacs 25.1 should sets its encoding
to UTF-8 in the file-local variables section near its end.  Do you
have that in your emacs.info?

> If I select the text and write the region to a file, I can open that file and Emacs renders it correctly. I tried to
> remedy this by adding the following to my init file (and commenting out the rest of the file):
> 
> (prefer-coding-system 'utf-8)
> (set-default-coding-systems 'utf-8)
> (set-terminal-coding-system 'utf-8)
> (set-keyboard-coding-system 'utf-8)
> (setq default-buffer-file-coding-system 'utf-8)

This should not be required for reading UTF-8 encoded Info manuals.

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Eli Zaretskii]

[Please keep the bug address on the CC list.]

> From: ivan <ivan.brennan@gmail.com>
> Date: Fri, 7 Oct 2016 22:12:40 -0400
> 
> Yes, my emacs.info file has the following local variables section:
> 
> Local Variables:
> coding: UTF-8
> End:

It should be "utf-8", in lower case.  How come it's in upper case on
your system?

Did you produce the Info files yourself, or did they come with some
package built elsewhere?

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[ivan]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

>
> It should be "utf-8", in lower case.  How come it's in upper case on
> your system?
>
> Did you produce the Info files yourself, or did they come with some
> package built elsewhere?
>

I used homebrew to build Emacs, so I believe the info files were generated
locally during the build process, but I didn't do anything special in that
regard.

I've been trying out two different build formulas,
1. https://github.com/d12frosted/homebrew-emacs-plus
    (just GNU Emacs with some build options)
2. https://github.com/railwaycat/homebrew-emacsmacport
    (GNU Emacs with some source code modifications for OSX)

Both builds have the uppercase "UTF-8" local variable in their info files,
and both exhibit the behavior I described.

[-- Attachment #2: Type: text/html, Size: 1476 bytes --]

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Eli Zaretskii]

> From: ivan <ivan.brennan@gmail.com>
> Date: Sat, 8 Oct 2016 08:31:33 -0400
> Cc: 24637@debbugs.gnu.org
> 
>  It should be "utf-8", in lower case. How come it's in upper case on
>  your system?
> 
>  Did you produce the Info files yourself, or did they come with some
>  package built elsewhere?
> 
> I used homebrew to build Emacs, so I believe the info files were generated locally during the build process,
> but I didn't do anything special in that regard.
> 
> I've been trying out two different build formulas,
> 1. https://github.com/d12frosted/homebrew-emacs-plus
> (just GNU Emacs with some build options)
> 2. https://github.com/railwaycat/homebrew-emacsmacport
> (GNU Emacs with some source code modifications for OSX)
> 
> Both builds have the uppercase "UTF-8" local variable in their info files, and both exhibit the behavior I
> described.

If you change that to lower-case, does the problem go away?

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[ivan]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

Sorry, I should have mentioned in my last response -- I tried changing them
to lowercase, but that didn't fix the problem.

On Sat, Oct 8, 2016 at 10:24 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: ivan <ivan.brennan@gmail.com>
> > Date: Sat, 8 Oct 2016 08:31:33 -0400
> > Cc: 24637@debbugs.gnu.org
> >
> >  It should be "utf-8", in lower case. How come it's in upper case on
> >  your system?
> >
> >  Did you produce the Info files yourself, or did they come with some
> >  package built elsewhere?
> >
> > I used homebrew to build Emacs, so I believe the info files were
> generated locally during the build process,
> > but I didn't do anything special in that regard.
> >
> > I've been trying out two different build formulas,
> > 1. https://github.com/d12frosted/homebrew-emacs-plus
> > (just GNU Emacs with some build options)
> > 2. https://github.com/railwaycat/homebrew-emacsmacport
> > (GNU Emacs with some source code modifications for OSX)
> >
> > Both builds have the uppercase "UTF-8" local variable in their info
> files, and both exhibit the behavior I
> > described.
>
> If you change that to lower-case, does the problem go away?
>

[-- Attachment #2: Type: text/html, Size: 1943 bytes --]

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Eli Zaretskii]

> From: ivan <ivan.brennan@gmail.com>
> Date: Sat, 8 Oct 2016 11:36:02 -0400
> Cc: 24637@debbugs.gnu.org
> 
> Sorry, I should have mentioned in my last response -- I tried changing them to lowercase, but that didn't fix the
> problem.

Than I'm out of ideas.

Does anyone else see this on OS X (or on any other OS)?

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[John Mastro]

Eli Zaretskii <eliz@gnu.org> wrote:
>> From: ivan <ivan.brennan@gmail.com>
>> Date: Sat, 8 Oct 2016 11:36:02 -0400
>> Cc: 24637@debbugs.gnu.org
>>
>> Sorry, I should have mentioned in my last response -- I tried changing them to lowercase, but that didn't fix the
>> problem.
>
> Than I'm out of ideas.
>
> Does anyone else see this on OS X (or on any other OS)?

I'm unable to reproduce with Emacs 25.1 compiled via Homebrew on MacOS
Sierra.

When I follow the recipe in the report, I see the correct glyphs, and
"utf-8" is lowercase as expected in the local variables section of
emacs.info.

Not sure if this is relevant, but I gave Homebrew the following
arguments when building: "--with-cocoa --with-gnutls --with-imagemagic".

        John

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]


() ivan <ivan.brennan@gmail.com>
() Sat, 8 Oct 2016 11:36:02 -0400

   changing them to lowercase, but that didn't fix the problem.

Did you restart Emacs?  I think simply ‘q’uitting out of Info
does not completely clear the various buffers / caches.

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (type via)
   (case type
     (technical (eq 'mailing-list via))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Alan Third]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: ivan <ivan.brennan@gmail.com>
>> Date: Sat, 8 Oct 2016 11:36:02 -0400
>> Cc: 24637@debbugs.gnu.org
>> 
>> Sorry, I should have mentioned in my last response -- I tried changing them to lowercase, but that didn't fix the
>> problem.
>
> Than I'm out of ideas.
>
> Does anyone else see this on OS X (or on any other OS)?

Just checked and I get the incorrect behaviour with the upper-case UTF-8
in emacs.info's "Local Variables:" section.

I built Emacs from git myself on OS X.

Now I think about it, I'm sure I've seen error messages regarding UTF-8
being invalid when building... Yup:

./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
  GEN      ../../info/eshell.info
./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
  GEN      ../../info/eudc.info
./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
  GEN      ../../info/efaq.info
./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
  GEN      ../../info/eww.info
./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
  GEN      ../../info/flymake.info

There are quite a lot of them. I guess one for each info file.

The only place I can think of right now that has this in upper case
that might affect the build is:

    breton:/Users/alan/src/emacs/master>echo $LANG
    en_GB.UTF-8

-- 
Alan Third

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Eli Zaretskii]

> From: Alan Third <alan@idiocy.org>
> Cc: ivan <ivan.brennan@gmail.com>,  24637@debbugs.gnu.org
> Date: Mon, 19 Dec 2016 11:27:56 +0000
> 
> > Does anyone else see this on OS X (or on any other OS)?
> 
> Just checked and I get the incorrect behaviour with the upper-case UTF-8
> in emacs.info's "Local Variables:" section.

Do you also have an old Texinfo?

> Now I think about it, I'm sure I've seen error messages regarding UTF-8
> being invalid when building... Yup:
> 
> ./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
>   GEN      ../../info/eshell.info
> ./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
>   GEN      ../../info/eudc.info
> ./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
>   GEN      ../../info/efaq.info
> ./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
>   GEN      ../../info/eww.info
> ./../emacs/docstyle.texi:3: warning: unrecognized encoding name `UTF-8'.
>   GEN      ../../info/flymake.info

The name is utf-8, in lower case.  Upper-case will not be recognized.
Coding system names are case-sensitive.

> The only place I can think of right now that has this in upper case
> that might affect the build is:
> 
>     breton:/Users/alan/src/emacs/master>echo $LANG
>     en_GB.UTF-8

That shouldn't matter.  The encoding of an Info file is taken from the
@documentencoding directive in the Texinfo source.

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Alan Third]

On Mon, Dec 19, 2016 at 06:18:02PM +0200, Eli Zaretskii wrote:
> > From: Alan Third <alan@idiocy.org>
> > Cc: ivan <ivan.brennan@gmail.com>,  24637@debbugs.gnu.org
> > Date: Mon, 19 Dec 2016 11:27:56 +0000
> > 
> > > Does anyone else see this on OS X (or on any other OS)?
> > 
> > Just checked and I get the incorrect behaviour with the upper-case UTF-8
> > in emacs.info's "Local Variables:" section.
> 
> Do you also have an old Texinfo?

Yup, that’s it exactly. macOS 10.12 comes with Texinfo 4.8.

As far as I can tell ./configure is looking for any version above 4.6.
I guess this is wrong, or the macOS version is broken.

For reference, I used homebrew to install 6.3 and this fixed it:

    brew install texinfo
    brew link texinfo --force

There are some warnings about linking in the upgraded version, but I
can’t really see what difference it makes, and I’m not sure whether
people building Emacs within homebrew itself will need to perform that
step.
-- 
Alan Third

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Eli Zaretskii]

> Date: Mon, 19 Dec 2016 17:06:53 +0000
> From: Alan Third <alan@idiocy.org>
> Cc: ivan.brennan@gmail.com, 24637@debbugs.gnu.org
> 
> > Do you also have an old Texinfo?
> 
> Yup, that’s it exactly. macOS 10.12 comes with Texinfo 4.8.
> 
> As far as I can tell ./configure is looking for any version above 4.6.
> I guess this is wrong, or the macOS version is broken.

We bumped the requirement lately, but only on master.

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 19 Dec 2016 17:06:53 +0000
>> From: Alan Third <alan@idiocy.org>
>> Cc: ivan.brennan@gmail.com, 24637@debbugs.gnu.org
>> 
>> > Do you also have an old Texinfo?
>> 
>> Yup, that’s it exactly. macOS 10.12 comes with Texinfo 4.8.
>> 
>> As far as I can tell ./configure is looking for any version above 4.6.
>> I guess this is wrong, or the macOS version is broken.
>
> We bumped the requirement lately, but only on master.

Should we do that for emacs-25 too?

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: Alan Third <alan@idiocy.org>,  24637@debbugs.gnu.org,  ivan.brennan@gmail.com
> Date: Thu, 02 Feb 2017 22:43:39 -0500
> 
> >> > Do you also have an old Texinfo?
> >> 
> >> Yup, that’s it exactly. macOS 10.12 comes with Texinfo 4.8.
> >> 
> >> As far as I can tell ./configure is looking for any version above 4.6.
> >> I guess this is wrong, or the macOS version is broken.
> >
> > We bumped the requirement lately, but only on master.
> 
> Should we do that for emacs-25 too?

I don't want to make any substantial changes on the release branch at
this time.  We are supposed to have RC1 shortly.

Thanks.

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Stefan Kangas]

Alan Third <alan@idiocy.org> writes:

> On Mon, Dec 19, 2016 at 06:18:02PM +0200, Eli Zaretskii wrote:
>> > From: Alan Third <alan@idiocy.org>
>> > Cc: ivan <ivan.brennan@gmail.com>,  24637@debbugs.gnu.org
>> > Date: Mon, 19 Dec 2016 11:27:56 +0000
>> > 
>> > > Does anyone else see this on OS X (or on any other OS)?
>> > 
>> > Just checked and I get the incorrect behaviour with the upper-case UTF-8
>> > in emacs.info's "Local Variables:" section.
>> 
>> Do you also have an old Texinfo?
>
> Yup, that’s it exactly. macOS 10.12 comes with Texinfo 4.8.
>
> As far as I can tell ./configure is looking for any version above 4.6.
> I guess this is wrong, or the macOS version is broken.
>
> For reference, I used homebrew to install 6.3 and this fixed it:
>
>     brew install texinfo
>     brew link texinfo --force
>
> There are some warnings about linking in the upgraded version, but I
> can’t really see what difference it makes, and I’m not sure whether
> people building Emacs within homebrew itself will need to perform that
> step.

If I read this correctly, the problem is with an old version of
texinfo (perhaps only on macOS).  If that is the case, this is not a
bug in Emacs and should be closed?

Best regards,
Stefan Kangas

bug#24637: 25.1; non-ascii chars fail to render in Info buffer

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> Alan Third <alan@idiocy.org> writes:
>
>> On Mon, Dec 19, 2016 at 06:18:02PM +0200, Eli Zaretskii wrote:
>>> > From: Alan Third <alan@idiocy.org>
>>> > Cc: ivan <ivan.brennan@gmail.com>,  24637@debbugs.gnu.org
>>> > Date: Mon, 19 Dec 2016 11:27:56 +0000
>>> > 
>>> > > Does anyone else see this on OS X (or on any other OS)?
>>> > 
>>> > Just checked and I get the incorrect behaviour with the upper-case UTF-8
>>> > in emacs.info's "Local Variables:" section.
>>> 
>>> Do you also have an old Texinfo?
>>
>> Yup, that’s it exactly. macOS 10.12 comes with Texinfo 4.8.
>>
>> As far as I can tell ./configure is looking for any version above 4.6.
>> I guess this is wrong, or the macOS version is broken.
>>
>> For reference, I used homebrew to install 6.3 and this fixed it:
>>
>>     brew install texinfo
>>     brew link texinfo --force
>>
>> There are some warnings about linking in the upgraded version, but I
>> can’t really see what difference it makes, and I’m not sure whether
>> people building Emacs within homebrew itself will need to perform that
>> step.
>
> If I read this correctly, the problem is with an old version of
> texinfo (perhaps only on macOS).  If that is the case, this is not a
> bug in Emacs and should be closed?

More information was requested, but none was given within 5 weeks, so
I'm closing this bug.  If this is still an issue, please reply to this
email (use "Reply to all" in your email client) and we can reopen the
bug report.

Best regards,
Stefan Kangas