23.0.60; Incorrect fontification for man in utf8 environment

23.0.60; Incorrect fontification for man in utf8 environment

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 180 bytes --]

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

LANG=en_US.utf8 emacs -Q
M-x man RET man RET

The resulting man page looks like this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/plain, Size: 779 bytes --]

MAN(1)                        Manual pager utils                        MAN(1)

NAME
       man − an interface to the on‐line reference manuals

SYNOPSIS
       man  [âˆâˆ’c|âˆâˆ’w|âˆâˆ’tZ] [âˆâˆ’H[browser]] [âˆâˆ’T[device]] [âˆâˆ’X[dpi]] [âˆâˆ’adhu7V] [âˆâˆ’i|âˆâˆ’I]
       [âˆâˆ’m system[,...]] [âˆâˆ’L locale] [âˆâˆ’p  string]  [âˆâˆ’C  file]  [âˆâˆ’M  path]  [âˆâˆ’P
       pager]  [âˆâˆ’r  prompt]  [âˆâˆ’S  list] [âˆâˆ’e extension] [âˆâˆ’âˆâˆ’warnings [warnings]]
       [[section] page ...] ...
       man âˆâˆ’l [âˆâˆ’7] [âˆâˆ’tZ] [âˆâˆ’H[browser]] [âˆâˆ’T[device]] [âˆâˆ’X[dpi]] [âˆâˆ’p string]  [âˆâˆ’P
       pager] [âˆâˆ’r prompt] [âˆâˆ’âˆâˆ’warnings[warnings]] file ...
       man âˆâˆ’k [apropos options] regexp ...
       man âˆâˆ’f [whatis options] page ...

[-- Attachment #3: Type: text/plain, Size: 52 bytes --]


The correct output, which occurs when LANG=C, is:


[-- Attachment #4: Type: text/plain, Size: 643 bytes --]

MAN(1)                        Manual pager utils                        MAN(1)

NAME
       man - an interface to the on-line reference manuals

SYNOPSIS
       man  [-c|-w|-tZ] [-H[browser]] [-T[device]] [-X[dpi]] [-adhu7V] [-i|-I]
       [-m system[,...]] [-L locale] [-p  string]  [-C  file]  [-M  path]  [-P
       pager]  [-r  prompt]  [-S  list] [-e extension] [--warnings [warnings]]
       [[section] page ...] ...
       man -l [-7] [-tZ] [-H[browser]] [-T[device]] [-X[dpi]] [-p string]  [-P
       pager] [-r prompt] [--warnings[warnings]] file ...
       man -k [apropos options] regexp ...
       man -f [whatis options] page ...

[-- Attachment #5: Type: text/plain, Size: 2454 bytes --]



In GNU Emacs 23.0.60.10 (i686-pc-linux-gnu, GTK+ Version 2.12.9)
 of 2008-04-01 on maru
Windowing system distributor `The X.Org Foundation', version 11.0.10400090
configured using `configure  '--without-toolkit-scroll-bars' '--with-dbus' '--enable-font-backend''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
  display-time-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent input:
M-x s h e l l <return> M-x m a n <return> m o u n t 
<return> m o u n t <return> M-x f i n d - f u n <tab> 
<return> m a n <return> <escape> < C-s c o d i n g 
- s t C-n C-n C-n C-n M-v C-v C-n C-p C-h v <return> 
C-x 1 C-n C-h v <return> C-x 1 C-x 4 b <return> C-x 
b <return> C-x b * m a <tab> <backspace> <backspace> 
M a <tab> <return> C-x o C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-x 1 C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-p C-p C-p M-x m a n <return> g r o f f <return> 
C-x o C-s N O _ <backspace> <backspace> <backspace> 
S G R C-a C-x o M-x m a n <return> m a n <return> C-x 
o C-s s g r C-g C-g C-g C-x o C-x 1 C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-x b <return> 
C-x k M-x r e p o r t - e m <tab> <return> I n f <backspace> 
c o r r e c t SPC f o n t i t <backspace> f i c v \ 
<backspace> <backspace> a t i o n SPC f o r SPC m a 
n SPC i n SPC u t f 8 SPC e n v i r o n m e n t <return> 
C-c C-k y e s <return> M-x r e p o r t - e m <tab> 
<return>

Recent messages:
Type C-x 1 to delete the help window.
Invoking man groff in the background
Please wait: formatting the groff man page...
groff man page formatted
Mark saved where search started
Invoking man man in the background
Please wait: formatting the man man page...
man man page formatted
Don't touch it!  It's the History Eraser Button, you fool! [2 times]
Parsing /home/md5i/.mailrc... done

-- 
Michael Welsh Duggan
(md5i@cs.cmu.edu)

Re: 23.0.60; Incorrect fontification for man in utf8 environment

[Chong Yidong]

> LANG=en_US.utf8 emacs -Q
> M-x man RET man RET
>
> The resulting man page looks like this:
>
> MAN(1)                        Manual pager utils                        MAN(1)
>
> NAME
>        man − an interface to the on‐line reference manuals
>
> SYNOPSIS
>        man  [?−c|?−w|?−tZ] [?−H[browser]] [?−T[device]] [?−X[dpi]] ...

I can't reproduce this, and my LANG is en_US.UTF-8.  Can anyone on this
list reproduce it?  If not, maybe it's been fixed in the meantime.

Re: 23.0.60; Incorrect fontification for man in utf8 environment

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

Chong Yidong <cyd@stupidchicken.com> writes:

>> LANG=en_US.utf8 emacs -Q
>> M-x man RET man RET
>>
>> The resulting man page looks like this:
>>
>> MAN(1)                        Manual pager utils                        MAN(1)
>>
>> NAME
>>        man − an interface to the on‐line reference manuals
>>
>> SYNOPSIS
>>        man  [?−c|?−w|?−tZ] [?−H[browser]] [?−T[device]] [?−X[dpi]] ...
>
> I can't reproduce this, and my LANG is en_US.UTF-8.  Can anyone on this
> list reproduce it?  If not, maybe it's been fixed in the meantime.

I just tried with the latest CVS sources.  Same problem.  Here is an
image:


[-- Attachment #2: man-problem.png --]
[-- Type: image/png, Size: 21315 bytes --]

[-- Attachment #3: Type: text/plain, Size: 44 bytes --]


-- 
Michael Welsh Duggan
(md5i@cs.cmu.edu)

Re: 23.0.60; Incorrect fontification for man in utf8 environment

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 123 bytes --]

Some more research on this:

The resulting file from:

 TERM=dumb GROFF_NO_SGR=1 man man > /tmp/man.out

on my system is:


[-- Attachment #2: man.out --]
[-- Type: application/octet-stream, Size: 31403 bytes --]

[-- Attachment #3: Type: text/plain, Size: 544 bytes --]


Looking at this file using hexl-find-file, one can see that formatting
codes are inserted in the output.  See from byte 0xad for example.

The coding system is set in man.el by:

	    (coding-system-for-read
	     (if default-enable-multibyte-characters
		 locale-coding-system 'raw-text-unix))

So the coding sytem is utf-8-unix on my system.  The aforementioned
formatting codes seem as if they are being interpreted as utf8-encoded
characters, causing the output from earlier in this bug report.

-- 
Michael Welsh Duggan
(md5i@cs.cmu.edu)

Re: 23.0.60; Incorrect fontification for man in utf8 environment

[Michael Welsh Duggan]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <md5i@cs.cmu.edu> writes:
>
>> Chong Yidong <cyd@stupidchicken.com> writes:
>>
>>>> LANG=en_US.utf8 emacs -Q
>>>> M-x man RET man RET
>
> [...]
>
>>> I can't reproduce this, and my LANG is en_US.UTF-8.  Can anyone on this
>>> list reproduce it?  If not, maybe it's been fixed in the meantime.
>>
>> I just tried with the latest CVS sources.  Same problem.  Here is an
>> image:
>
> I've just tried this with Emacs 24, and I'm unable to reproduce.  Has
> this bug gone away?

It seems to have.

-- 
Michael Welsh Duggan
(md5i@md5i.com)