Re: coding problem in article display with gnus 5.13 incl. in emacs cvs

Re: coding problem in article display with gnus 5.13 incl. in emacs cvs

[Reiner Steib]

On Tue, Apr 08 2008, Patrick Drechsler wrote:

> after updating GNU/Emacs CVS a few days ago I noticed that the article
> buffer sometimes does not display umlaut characters correctly anymore.

I'm not aware of any changes in Gnus, thus most probably a change in
Emacs is the reason.  (Cc-ing emacs-devel, keeping full citation.)

> Symptoms: Either the article buffer displays (A) the character's control
> sequence (ie "ä" -> "\344") or (B) it skips the character it can not
> display. This did not happen in previous versions of Gnus v5.13 shipped
> with Emacs CVS.
>
> To reproduce (A):
>
> Step 1/2: Send yourself a mail with another mail client (for example
> Thunderbird) with the following content (do not copy the umlauts from
> the example below: Type them in i.e. Thunderbird!):
>
> here are some german umlauts:
>
> ae: ä
> oe: ö
> ue: ü
> ss: ß
>
> end of test
>
> Step 2/2: This message is now in my "Sent" folder. It is displayed
> incorrectly in Gnus (Below I replaced the control characters by the
> string displayed):
>
> From: Patrick Drechsler <patrick@XXX.XX>
> Subject: test umlauts
> To: Patrick Drechsler <patrickXdrechsler@XXX.XXX>
> User-Agent: Thunderbird 1.5.0.14ubu (X11/20080306)
>
> here are some german umlauts:
>
> ae: \344
> oe: \366
> ue: \374
> ss: \337
>
> end of test

Does the problem only appear in Gcc-ed messages or in usual mail/news
messages as well?  If the later, could you send such a message to 
gmane-test@hawk.netfonds.no (gmane.test).

> Placing the cursor on "\344" and `M-x describe-char' results in:
>
> ,----
> |         character: \344 (4194276, #o17777744, #x3fffe4)
> | preferred charset: eight-bit (Raw bytes 0-255)
> |        code point: 0xE4
> |            syntax: w 	which means: word
> |       buffer code: #xE4
> |         file code: not encodable by coding system undecided-unix
> |           display: no font available
> | 
> | Character code properties are not shown: customize what to show
> | 
> | There are text properties here:
> |   auto-composed        t
> `---- 
>
> Here is the output of `M-x describe-char' from the ä character in second
> sentence:
>
> ,----
> |         character: ä (228, #o344, #xe4)
> | preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
> |        code point: 0xE4
> |            syntax: w 	which means: word
> |          category: j:Japanese l:Latin
> |       buffer code: #xC3 #xA4
> |         file code: #xC3 #xA4 (encoded by coding system utf-8-emacs-unix)
> |           display: by this font (glyph code)
> |      bitstream vera sans mono:pixelsize=13:foundry=bitstream:weight=regular:slant=roman:width=normal (#x6C)
> `----
>
> (Don't ask me why the category indicates "j:Japanese")
>
> Concerning (B):
>
> On other occations I noticed that displayed umlauts are deleted or
> replaced by a space.
>
> Here is an example:
>
> ,----
> | List-Id: Linux <linux.listserv.heise.de>
> | To: patrickXXX@XXX.XX
> | From: Linux <newsletter@listserv.heise.de>
> | Date: Mon, 07 Apr 2008 04:02:02 +0200
> | Content-Type: multipart/alternative; boundary="-----b007da1ee42"
> | Content-Transfer-Encoding: 8bit
> | MIME-Version: 1.0
> | Subject: heise online News Linux 07.04.2008
> | Message-Id: <E1JiiJi-00083q-2g.05@web.heise.de>
> | [...] 
> |
> | *  Nachrichten berblick zum Thema Linux der vergangenen sieben Tage          *
> | *
> |    Benchmark-Suite f r Linux
> | 
> |    Phoronix hat eine umfassende Benchmark-Suite f r Linux ver ffentlicht.
> | [...] 
> `----
>
> Can other people using Gnus (not "No Gnus") with current Gnu Emacs CVS
> confirm this behaviour?

FTR: Gnus 5.13 is the same as "No Gnus", modulo delays in
synchronization.

> Going through the Gnus settings below, the only thing that catches my
> eye is "mm-charset-synonym-alist". Searching the Gnus manual for the
> terms "charset" results in section 3.20 Charsets. But I have trouble
> understanding it.

`mm-charset-synonym-alist' should be irrelevant.

> Can anybody give me a pointer which variable(s) need to be adopted?
>
> Previous versions of Gnus v5.13 included with Emacs CVS displayed the
> article buffer correctly.

It would be very good if you could identify the change causing the
problem using bisection (specifying the date to cvs).

> TIA,
>
> Patrick
>
> ,----[ Emacs version and settings: ]
> | In GNU Emacs 23.0.60.2 (i686-pc-linux-gnu, GTK+ Version 2.8.20)
> |  of 2008-04-04 on golem
> | Windowing system distributor `The X.Org Foundation', version 11.0.70000000
> | configured using `configure  '--prefix=/home/patrick/prg/stow/emacs-devel' '--with-xft' '--with-x-toolkit=gtk''
> | 
> | Important settings:
> |   value of $LC_ALL: nil
> |   value of $LC_COLLATE: nil
> |   value of $LC_CTYPE: nil
> |   value of $LC_MESSAGES: nil
> |   value of $LC_MONETARY: nil
> |   value of $LC_NUMERIC: nil
> |   value of $LC_TIME: nil
> |   value of $LANG: de_DE.UTF-8
> |   value of $XMODIFIERS: nil
> |   locale-coding-system: utf-8-unix
> |   default-enable-multibyte-characters: t
> | 
> | Major mode: Message
> | 
> | Minor modes in effect:
> |   gnus-message-citation-mode: t
> |   mml-mode: t
> |   delete-selection-mode: t
> |   show-paren-mode: t
> |   savehist-mode: t
> |   pc-selection-mode: t
> |   iswitchb-mode: t
> |   display-time-mode: t
> |   which-function-mode: t
> |   shell-dirtrack-mode: t
> |   tooltip-mode: t
> |   mouse-wheel-mode: t
> |   menu-bar-mode: t
> |   file-name-shadow-mode: t
> |   global-font-lock-mode: t
> |   font-lock-mode: t
> |   blink-cursor-mode: t
> |   global-auto-composition-mode: t
> |   auto-composition-mode: t
> |   auto-compression-mode: t
> |   column-number-mode: t
> |   line-number-mode: t
> |   transient-mark-mode: t
> |   abbrev-mode: t
> `----
>
> ,----[ Gnus settings ]
> | (setq gnus-default-nntp-server "")
> | (setq gnus-select-method
> |       '(nntp "news.myserver.foo"))
> | (setq gnus-message-archive-group
> |       '((if
> | 	    (message-news-p)
> | 	    "nnimap+mail.mymailserver.bla:INBOX.sent.news" "nnimap+mail.mymailserver.bla:INBOX.Sent")))
> | (setq gnus-secondary-select-methods
> |       '((nnml "private")
> | 	(nntp "news.gmane.org")
> | 	(nnimap "mail.mymailserver.bla"
> | 		(nnimap-list-pattern
> | 		 ("*")))))
> | (setq gnus-group-change-level-function 'gnus-topic-change-level)
> | (setq gnus-summary-expunge-below -999)
> | (setq gnus-summary-mode-hook
> |       '(gnus-agent-mode))
> | (setq gnus-exit-group-hook
> |       '(gnus-uu-delete-work-dir gnus-uu-clean-up))
> | (setq gnus-group-line-format "%M%S%p%P%5y:%B%(%G%)%l %O\n")
> | (setq gnus-group-catchup-group-hook
> |       '(gnus-topic-update-topic))
> | (setq gnus-exit-gnus-hook
> |       '(mm-destroy-postponed-undisplay-list))
> | (setq gnus-visible-headers "^From:\\|^Newsgroups:\\|^Subject:\\|^Date:\\|^Followup-To:\\|^Reply-To:\\|^Organization:\\|^Summary:\\|^Keywords:\\|^To:\\|^[BGF]?Cc:\\|^Posted-To:\\|^Mail-Copies-To:\\|^Mail-Followup-To:\\|^Apparently-To:\\|^Gnus-Warning:\\|^Resent-From:\\|^X-Sent:\\|^X-Newsreader:\\|^X-Mailer:\\|^User-Agent:\\|^X-User-Agent:\\|^X-Injected-Via-Gmane:\\|^Message-ID:")
> | (setq gnus-saved-headers "^From:\\|^Newsgroups:\\|^Subject:\\|^Date:\\|^Followup-To:\\|^Reply-To:\\|^Organization:\\|^Summary:\\|^Keywords:\\|^To:\\|^[BGF]?Cc:\\|^Posted-To:\\|^Mail-Copies-To:\\|^Mail-Followup-To:\\|^Apparently-To:\\|^Gnus-Warning:\\|^Resent-From:\\|^X-Sent:")
> | (setq gnus-article-prepare-hook
> |       '(bbdb/gnus-pop-up-bbdb-buffer))
> | (setq gnus-init-file "~/.emacs.d/.gnus")
> | (setq gnus-startup-hook
> |       '(bbdb-insinuate-gnus))
> | (setq gnus-setup-news-hook
> |       '(gnus-agent-queue-setup gnus-fixup-nnimap-unread-after-getting-new-news))
> | (setq gnus-save-newsrc-hook
> |       '(bbdb-offer-save))
> | (setq gnus-group-mode-hook
> |       '(gnus-agent-mode gnus-topic-mode))
> | (setq gnus-posting-styles
> |       '((".*"
> | 	 (organization "none")
> | 	 (Face
> | 	  '(gnus-convert-png-to-face "~/.emacs.d/faces/ant.png")))
> | 	(".*gmane.comp.lang.r.general"
> | 	 (signature-file "~/.emacs.d/mailsignature/sig-workcam-r")
> | 	 (Face
> | 	  '(gnus-convert-png-to-face "~/.emacs.d/faces/ant.png")))))
> | (setq gnus-home-score-file "~/.emacs.d/total-score-file.SCORE")
> | (setq gnus-server-mode-hook
> |       '(gnus-agent-mode))
> | (setq mm-charset-synonym-alist
> |       '((ibm866 . cp866)
> | 	(unicode . utf-16-le)
> | 	(ks_c_5601-1987 . cp949)
> | 	(windows-31j . cp932)
> | 	(iso8859-1 . iso-8859-1)))
> | (setq message-send-mail-function 'smtpmail-send-it)
> | (setq message-post-method
> |       '(nntp "news"))
> | (setq message-setup-hook
> |       '(bbdb-define-all-aliases bbdb-insinuate-message))
> | (setq message-mode-hook
> |       '(#[nil "\302\030\303	!)\207"
> | 	      [gnus-article-copy gnus-setup-message-group nil gnus-configure-posting-styles]
> | 	      2]
> | 	#[nil "\302 \211\020\211\021\207"
> | 	      [message-mailer message-newsreader gnus-extended-version]
> | 	      2]))
> | (setq message-header-setup-hook
> |       '(gnus-inews-insert-archive-gcc gnus-inews-insert-gcc))
> | (setq message-signature 'randomsig-signature)
> `----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: coding problem in article display with gnus 5.13 incl. in emacs cvs

[Patrick Drechsler]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> On Tue, Apr 08 2008, Patrick Drechsler wrote:
>
>> after updating GNU/Emacs CVS a few days ago I noticed that the
>> article buffer sometimes does not display umlaut characters correctly
>> anymore.
>
> I'm not aware of any changes in Gnus, thus most probably a change in
> Emacs is the reason.  (Cc-ing emacs-devel, keeping full citation.)
>
>> Symptoms: Either the article buffer displays (A) the character's
>> control sequence (ie "ä" -> "\344") or (B) it skips the character it
>> can not display. This did not happen in previous versions of Gnus
>> v5.13 shipped with Emacs CVS.
>>
>> To reproduce (A):
[...snipped...]

> Does the problem only appear in Gcc-ed messages or in usual mail/news
> messages as well?  If the later, could you send such a message to
> gmane-test@hawk.netfonds.no (gmane.test).

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/plain, Size: 1328 bytes --]


If I send a test message to gmane.test[1] using Thunderbird, and then
view the message with Gnus, the characters are displayed correctly when
visiting the group. On the other hand Gnus displays the message
incorrectly in my "Sent" folder:
,----
| User-Agent: Thunderbird 1.5.0.14ubu (X11/20080306)
| MIME-Version: 1.0
| Newsgroups: gmane.test
| Subject: test
| Content-Type: text/plain; charset=UTF-8; format=flowed
| Content-Transfer-Encoding: 8bit
| [...]
|         character: ä (4194276, #o17777744, #x3fffe4)
| preferred charset: eight-bit (Raw bytes 0-255)
|        code point: 0xE4
|            syntax: w 	which means: word
|       buffer code: #xE4
|         file code: not encodable by coding system windows-1252-unix
|           display: no font available
`----

Also other "normal" Emails are displayed incorrectly. Example:
,----
| User-Agent: Thunderbird 2.0.0.12 (Windows/20080213)
| MIME-Version: 1.0
| Content-Type: text/plain; charset=UTF-8; format=flowed
| Content-Transfer-Encoding: 8bit
| [...]
|         character: ä (4194276, #o17777744, #x3fffe4)
| preferred charset: eight-bit (Raw bytes 0-255)
|        code point: 0xE4
|            syntax: w 	which means: word
|       buffer code: #xE4
|         file code: not encodable by coding system windows-1252-unix
|           display: no font available
`----

[-- Attachment #3: Type: text/plain, Size: 8458 bytes --]


>> Placing the cursor on "\344" and `M-x describe-char' results in:
>>
>> ,----
>> |         character: \344 (4194276, #o17777744, #x3fffe4)
>> | preferred charset: eight-bit (Raw bytes 0-255)
>> |        code point: 0xE4
>> |            syntax: w 	which means: word
>> |       buffer code: #xE4
>> |         file code: not encodable by coding system undecided-unix
>> |           display: no font available
>> | 
>> | Character code properties are not shown: customize what to show
>> | 
>> | There are text properties here:
>> |   auto-composed        t
>> `---- 
>>
>> Here is the output of `M-x describe-char' from the ä character in
>> second sentence:
>>
>> ,----
>> |         character: ä (228, #o344, #xe4)
>> | preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
>> |        code point: 0xE4
>> |            syntax: w 	which means: word
>> |          category: j:Japanese l:Latin
>> |       buffer code: #xC3 #xA4
>> |         file code: #xC3 #xA4 (encoded by coding system utf-8-emacs-unix)
>> |           display: by this font (glyph code)
>> |      bitstream vera sans mono:pixelsize=13:foundry=bitstream:weight=regular:slant=roman:width=normal (#x6C)
>> `----
>>
>> (Don't ask me why the category indicates "j:Japanese")
>>
>> Concerning (B):
>>
>> On other occations I noticed that displayed umlauts are deleted or
>> replaced by a space.
>>
>> Here is an example:
>>
>> ,----
>> | List-Id: Linux <linux.listserv.heise.de>
>> | To: patrickXXX@XXX.XX
>> | From: Linux <newsletter@listserv.heise.de>
>> | Date: Mon, 07 Apr 2008 04:02:02 +0200
>> | Content-Type: multipart/alternative; boundary="-----b007da1ee42"
>> | Content-Transfer-Encoding: 8bit
>> | MIME-Version: 1.0
>> | Subject: heise online News Linux 07.04.2008
>> | Message-Id: <E1JiiJi-00083q-2g.05@web.heise.de>
>> | [...] 
>> |
>> | *  Nachrichten berblick zum Thema Linux der vergangenen sieben Tage          *
>> | *
>> |    Benchmark-Suite f r Linux
>> | 
>> |    Phoronix hat eine umfassende Benchmark-Suite f r Linux ver ffentlicht.
>> | [...] 
>> `----
>>
>> Can other people using Gnus (not "No Gnus") with current Gnu Emacs
>> CVS confirm this behaviour?
>
> FTR: Gnus 5.13 is the same as "No Gnus", modulo delays in
> synchronization.

OK, thanks for correcting me.

>> Going through the Gnus settings below, the only thing that catches my
>> eye is "mm-charset-synonym-alist". Searching the Gnus manual for the
>> terms "charset" results in section 3.20 Charsets. But I have trouble
>> understanding it.
>
> `mm-charset-synonym-alist' should be irrelevant.

OK.

>> Can anybody give me a pointer which variable(s) need to be adopted?
>>
>> Previous versions of Gnus v5.13 included with Emacs CVS displayed the
>> article buffer correctly.
>
> It would be very good if you could identify the change causing the
> problem using bisection (specifying the date to cvs).

I have never done this, but I'll give a try this weekend. How do I check
out older cvs versions by date? This does not work:

--8<---------------cut here---------------start------------->8---
cvs -z3 \
    -D "2008-02-01 00:00:00" \
    -d:pserver:anonymous@cvs.savannah.gnu.org:/cvsroot/emacs \
    co emacs
--8<---------------cut here---------------end--------------->8---

Thanks

Patrick

>> ,----[ Emacs version and settings: ]
>> | In GNU Emacs 23.0.60.2 (i686-pc-linux-gnu, GTK+ Version 2.8.20)
>> |  of 2008-04-04 on golem
>> | Windowing system distributor `The X.Org Foundation', version 11.0.70000000
>> | configured using `configure  '--prefix=/home/patrick/prg/stow/emacs-devel' '--with-xft' '--with-x-toolkit=gtk''
>> | 
>> | Important settings:
>> |   value of $LC_ALL: nil
>> |   value of $LC_COLLATE: nil
>> |   value of $LC_CTYPE: nil
>> |   value of $LC_MESSAGES: nil
>> |   value of $LC_MONETARY: nil
>> |   value of $LC_NUMERIC: nil
>> |   value of $LC_TIME: nil
>> |   value of $LANG: de_DE.UTF-8
>> |   value of $XMODIFIERS: nil
>> |   locale-coding-system: utf-8-unix
>> |   default-enable-multibyte-characters: t
>> | 
>> | Major mode: Message
>> | 
>> | Minor modes in effect:
>> |   gnus-message-citation-mode: t
>> |   mml-mode: t
>> |   delete-selection-mode: t
>> |   show-paren-mode: t
>> |   savehist-mode: t
>> |   pc-selection-mode: t
>> |   iswitchb-mode: t
>> |   display-time-mode: t
>> |   which-function-mode: t
>> |   shell-dirtrack-mode: t
>> |   tooltip-mode: t
>> |   mouse-wheel-mode: t
>> |   menu-bar-mode: t
>> |   file-name-shadow-mode: t
>> |   global-font-lock-mode: t
>> |   font-lock-mode: t
>> |   blink-cursor-mode: t
>> |   global-auto-composition-mode: t
>> |   auto-composition-mode: t
>> |   auto-compression-mode: t
>> |   column-number-mode: t
>> |   line-number-mode: t
>> |   transient-mark-mode: t
>> |   abbrev-mode: t
>> `----
>>
>> ,----[ Gnus settings ]
>> | (setq gnus-default-nntp-server "")
>> | (setq gnus-select-method
>> |       '(nntp "news.myserver.foo"))
>> | (setq gnus-message-archive-group
>> |       '((if
>> | 	    (message-news-p)
>> | 	    "nnimap+mail.mymailserver.bla:INBOX.sent.news" "nnimap+mail.mymailserver.bla:INBOX.Sent")))
>> | (setq gnus-secondary-select-methods
>> |       '((nnml "private")
>> | 	(nntp "news.gmane.org")
>> | 	(nnimap "mail.mymailserver.bla"
>> | 		(nnimap-list-pattern
>> | 		 ("*")))))
>> | (setq gnus-group-change-level-function 'gnus-topic-change-level)
>> | (setq gnus-summary-expunge-below -999)
>> | (setq gnus-summary-mode-hook
>> |       '(gnus-agent-mode))
>> | (setq gnus-exit-group-hook
>> |       '(gnus-uu-delete-work-dir gnus-uu-clean-up))
>> | (setq gnus-group-line-format "%M%S%p%P%5y:%B%(%G%)%l %O\n")
>> | (setq gnus-group-catchup-group-hook
>> |       '(gnus-topic-update-topic))
>> | (setq gnus-exit-gnus-hook
>> |       '(mm-destroy-postponed-undisplay-list))
>> | (setq gnus-visible-headers "^From:\\|^Newsgroups:\\|^Subject:\\|^Date:\\|^Followup-To:\\|^Reply-To:\\|^Organization:\\|^Summary:\\|^Keywords:\\|^To:\\|^[BGF]?Cc:\\|^Posted-To:\\|^Mail-Copies-To:\\|^Mail-Followup-To:\\|^Apparently-To:\\|^Gnus-Warning:\\|^Resent-From:\\|^X-Sent:\\|^X-Newsreader:\\|^X-Mailer:\\|^User-Agent:\\|^X-User-Agent:\\|^X-Injected-Via-Gmane:\\|^Message-ID:")
>> | (setq gnus-saved-headers "^From:\\|^Newsgroups:\\|^Subject:\\|^Date:\\|^Followup-To:\\|^Reply-To:\\|^Organization:\\|^Summary:\\|^Keywords:\\|^To:\\|^[BGF]?Cc:\\|^Posted-To:\\|^Mail-Copies-To:\\|^Mail-Followup-To:\\|^Apparently-To:\\|^Gnus-Warning:\\|^Resent-From:\\|^X-Sent:")
>> | (setq gnus-article-prepare-hook
>> |       '(bbdb/gnus-pop-up-bbdb-buffer))
>> | (setq gnus-init-file "~/.emacs.d/.gnus")
>> | (setq gnus-startup-hook
>> |       '(bbdb-insinuate-gnus))
>> | (setq gnus-setup-news-hook
>> |       '(gnus-agent-queue-setup gnus-fixup-nnimap-unread-after-getting-new-news))
>> | (setq gnus-save-newsrc-hook
>> |       '(bbdb-offer-save))
>> | (setq gnus-group-mode-hook
>> |       '(gnus-agent-mode gnus-topic-mode))
>> | (setq gnus-posting-styles
>> |       '((".*"
>> | 	 (organization "none")
>> | 	 (Face
>> | 	  '(gnus-convert-png-to-face "~/.emacs.d/faces/ant.png")))
>> | 	(".*gmane.comp.lang.r.general"
>> | 	 (signature-file "~/.emacs.d/mailsignature/sig-workcam-r")
>> | 	 (Face
>> | 	  '(gnus-convert-png-to-face "~/.emacs.d/faces/ant.png")))))
>> | (setq gnus-home-score-file "~/.emacs.d/total-score-file.SCORE")
>> | (setq gnus-server-mode-hook
>> |       '(gnus-agent-mode))
>> | (setq mm-charset-synonym-alist
>> |       '((ibm866 . cp866)
>> | 	(unicode . utf-16-le)
>> | 	(ks_c_5601-1987 . cp949)
>> | 	(windows-31j . cp932)
>> | 	(iso8859-1 . iso-8859-1)))
>> | (setq message-send-mail-function 'smtpmail-send-it)
>> | (setq message-post-method
>> |       '(nntp "news"))
>> | (setq message-setup-hook
>> |       '(bbdb-define-all-aliases bbdb-insinuate-message))
>> | (setq message-mode-hook
>> |       '(#[nil "\302\030\303	!)\207"
>> | 	      [gnus-article-copy gnus-setup-message-group nil gnus-configure-posting-styles]
>> | 	      2]
>> | 	#[nil "\302 \211\020\211\021\207"
>> | 	      [message-mailer message-newsreader gnus-extended-version]
>> | 	      2]))
>> | (setq message-header-setup-hook
>> |       '(gnus-inews-insert-archive-gcc gnus-inews-insert-gcc))
>> | (setq message-signature 'randomsig-signature)
>> `----


Footnotes: 
[1] Archived-At: <http://permalink.gmane.org/gmane.test/4646>

-- 
For animals, the entire universe has been neatly divided into things
to (a) mate with, (b) eat, (c) run away from, and (d) rocks.
        (Terry Pratchett, Equal Rites)