notmuch-emacs won't display correctly quoted-printable iso-8859-1 mails

notmuch-emacs won't display correctly quoted-printable iso-8859-1 mails

[Olivier Berger]

Hi.

Here's a copy of a bug report I've just filed in Debian's bugtracker [0].

If some of you have an idea of whether there's a fix/workaround ?

Many thanks in advance.

Best regards,

P.S.: sent with Gnus to avoid messing with the encodings too much ;)

[0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=652359

Olivier Berger <olivier.berger@it-sudparis.eu> writes:

> Package: notmuch-emacs
> Version: 0.10.1-1
> Severity: normal
>
> Hi.
>
> I experience problems with communication with other people in french, when sending/responding to emails with notmuch.
>
> The following is the source of a mail sent with notmuch :
>
> MIME-Version: 1.0
> Content-Type: text/plain; charset=iso-8859-1
> Content-Transfer-Encoding: quoted-printable
>
> Du texte accentu=E9 pour =E7a ...
>
> =E0 la bonne heure !
> --=20
>
> Which is rendered as :
>
>
> Du texte accentué pour ça ...
>
> à la bonne heure !
> [ 6-line signature. Click/Enter to hide. ]

-- 
Olivier BERGER 
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)

Re: notmuch-emacs won't display correctly quoted-printable iso-8859-1 mails

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

On Fri, 16 Dec 2011 17:28:04 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:

> 
> Here's a copy of a bug report I've just filed in Debian's bugtracker [0].
> 
> If some of you have an idea of whether there's a fix/workaround ?
> 

Hi Olivier;

Can you try the following patch? If you apply it to git, you can use
"make debian-snapshot" to build new packages (assuming you have the
pre-reqs).

Or just patch the notmuch-query.el installed by notmuch-emacs and reload
it.

The search buffer will still be messed up, but if show is OK, that gives
a hint we are on the right track.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: recognize that notmuch will output utf8, regardless of locale --]
[-- Type: text/x-diff, Size: 537 bytes --]

diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
index d66baea..7390f82 100644
--- a/emacs/notmuch-query.el
+++ b/emacs/notmuch-query.el
@@ -38,7 +38,7 @@ is a possibly empty forest of replies.
     (setq args (append args search-terms))
     (with-temp-buffer
       (progn
-	(apply 'call-process (append (list notmuch-command nil (list t nil) nil) args))
+	(let ((coding-system-for-read 'utf-8)) (apply 'call-process (append (list notmuch-command nil (list t nil) nil) args)))
 	(goto-char (point-min))
 	(json-read)))))
 

Re: notmuch-emacs won't display correctly quoted-printable iso-8859-1 mails

[Olivier Berger]

On Sun, 18 Dec 2011 10:11:37 -0400, David Bremner <david@tethera.net> wrote:
> On Fri, 16 Dec 2011 17:28:04 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> 
> > 
> > Here's a copy of a bug report I've just filed in Debian's bugtracker [0].
> > 
> > If some of you have an idea of whether there's a fix/workaround ?
> > 
> 
> Hi Olivier;
> 
> Can you try the following patch? If you apply it to git, you can use
> "make debian-snapshot" to build new packages (assuming you have the
> pre-reqs).
> 
> Or just patch the notmuch-query.el installed by notmuch-emacs and reload
> it.
> 

I did that over notmuch-emacs 0.10.2-1 Debian package's version of
notmuch-query.el, but that doesn't seem change anything, unfortunately :
the modeline still is '-1:%*-' for the notmuch-show buffer, after
hitting RET over a message's line in a search result list :-(

Hope this helps.

Best regards,
-- 
Olivier BERGER 
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)

Re: notmuch-emacs won't display correctly quoted-printable iso-8859-1 mails

[Tomi Ollila]

On Sun, 18 Dec 2011 15:53:26 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> On Sun, 18 Dec 2011 10:11:37 -0400, David Bremner <david@tethera.net> wrote:
> > Hi Olivier;
> > 
> > Can you try the following patch? If you apply it to git, you can use
> > "make debian-snapshot" to build new packages (assuming you have the
> > pre-reqs).
> > 
> > Or just patch the notmuch-query.el installed by notmuch-emacs and reload
> > it.
> > 
> 
> I did that over notmuch-emacs 0.10.2-1 Debian package's version of
> notmuch-query.el, but that doesn't seem change anything, unfortunately :
> the modeline still is '-1:%*-' for the notmuch-show buffer, after
> hitting RET over a message's line in a search result list :-(

I tested the same on terminal configured for latin9 and LC_ALL=fi_FI@euro
and the change worked for me.

The buffer modeline is not supposed to change -- the change makes emacs
read incoming data encoded in utf-8 format (notmuch outputs everything
in utf-8). Before the change emacs expected (in your case) input data being
in latin1 format ("guessed" from your locale), but as input was in utf-8 the
conversion to emacs internal format went wrong.

When emacs displays something (buffer content, that is) it converts the
internal format to the encoding emacs window is using. 

So, my guess is you did something wrong when trying David's patch and
you did not get the change evaluated.

This what I did:

I opened emacs/notmuch-query.el to another emacs window while 
notmuch-hello open in another window.

Then I added line (coding-system-for-read 'utf-8) in line 35:

  (let  ((args '("show" "--format=json"))
         (json-object-type 'plist)
         (json-array-type 'list)
         (coding-system-for-read 'utf-8)
         (json-false 'nil))

then moved cursor to the end of line 44 which shows: (json-read)))))
(last line of that function) and entered c-x c-e
(eval-last-sexp) -- that re-evaluates the function definition.

And, as said, after that change my emails render correctly on
latin1 -terminal (as opposed those did not render correctly before)

> Hope this helps.

I hope that I'm right in my guess so we get forward easier... :)

> 
> Best regards,
> -- 
> Olivier BERGER 

Tomi

Re: notmuch-emacs won't display correctly quoted-printable iso-8859-1 mails

[Olivier Berger]

Hi.

(message previously sent privately, and resent to the list and BTS for reference)

On Sun, 18 Dec 2011 18:08:55 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> On Sun, 18 Dec 2011 15:53:26 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> > On Sun, 18 Dec 2011 10:11:37 -0400, David Bremner <david@tethera.net> wrote:
> > > Hi Olivier;
> > > 
> > > Can you try the following patch? If you apply it to git, you can use
> > > "make debian-snapshot" to build new packages (assuming you have the
> > > pre-reqs).
> > > 
> > > Or just patch the notmuch-query.el installed by notmuch-emacs and reload
> > > it.
> > > 
> > 
> > I did that over notmuch-emacs 0.10.2-1 Debian package's version of
> > notmuch-query.el, but that doesn't seem change anything, unfortunately :
> > the modeline still is '-1:%*-' for the notmuch-show buffer, after
> > hitting RET over a message's line in a search result list :-(
> 
> I tested the same on terminal configured for latin9 and LC_ALL=fi_FI@euro
> and the change worked for me.

Glad for you... but did it work before too, by any chance ?

Maybe it wasn't clear in my report, bug I'm using emacs23 in X, and not
in terminal. Which is different from your tests, AFAIU.

> 
> The buffer modeline is not supposed to change -- the change makes emacs
> read incoming data encoded in utf-8 format (notmuch outputs everything
> in utf-8). Before the change emacs expected (in your case) input data being
> in latin1 format ("guessed" from your locale), but as input was in utf-8 the
> conversion to emacs internal format went wrong.

My locale is : fr_FR.utf8 ... maybe you're guessing a bit too much, and
again, emacs runs in X... as for your modeline explanation, it's not
really clear I'm afraid. I have always had the impression that the
modeline should be starting with -U:... if I'm supposed to display
correctly some UTF-8 characters, which is not the case, hence the
problem. I don't know what else should happen. So AFAICT, the goal is to
make sure the buffer is indeed "opened" as UTF-8, or rendered as UTF-8,
although I couldn't tell how emacs does this all, to be able to
understand the patch correctly.

> 
> When emacs displays something (buffer content, that is) it converts the
> internal format to the encoding emacs window is using. 
> 
> So, my guess is you did something wrong when trying David's patch and
> you did not get the change evaluated.
> 

I don't think so.

> This what I did:
> 
> I opened emacs/notmuch-query.el to another emacs window while 
> notmuch-hello open in another window.
> 
> Then I added line (coding-system-for-read 'utf-8) in line 35:
> 
>   (let  ((args '("show" "--format=json"))
>          (json-object-type 'plist)
>          (json-array-type 'list)
>          (coding-system-for-read 'utf-8)
>          (json-false 'nil))
> 

Uh, is this really the patch suggested by David ?

Ain't it supposed to be :
  --- a/emacs/notmuch-query.el
  +++ b/emacs/notmuch-query.el
  @@ -38,7 +38,7 @@ is a possibly empty forest of replies.
       (setq args (append args search-terms))
       (with-temp-buffer
	 (progn
  -	(apply 'call-process (append (list notmuch-command nil (list t nil) nil) args))
  +	(let ((coding-system-for-read 'utf-8)) (apply 'call-process (append (list notmuch-command nil (list t nil) nil) args)))
	  (goto-char (point-min))
	  (json-read)))))

Anyway, I had quit emacs, then patched the file with the above patch,
and restarted it, so I don't know what could have gone wrong.

> then moved cursor to the end of line 44 which shows: (json-read)))))
> (last line of that function) and entered c-x c-e
> (eval-last-sexp) -- that re-evaluates the function definition.
> 
> And, as said, after that change my emails render correctly on
> latin1 -terminal (as opposed those did not render correctly before)
> 

OK, so maybe that's one fix for the terminal, but not yet complete for
the X/Gtk Emacs windows.

> > Hope this helps.
> 
> I hope that I'm right in my guess so we get forward easier... :)
> 

A bit too much guessing I'm afraid ;)

Thanks anyway for your help.

Best regards,

-- 
Olivier BERGER 
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)

Re: Bug#652359: notmuch-emacs won't display correctly quoted-printable iso-8859-1 mails

[David Bremner]

On Sun, 18 Dec 2011 15:53:26 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> I did that over notmuch-emacs 0.10.2-1 Debian package's version of
> notmuch-query.el, but that doesn't seem change anything, unfortunately :
> the modeline still is '-1:%*-' for the notmuch-show buffer, after
> hitting RET over a message's line in a search result list :-(

Just to be clear, is the message displayed correctly in notmuch-show
mode? The modeline is not supposed to change, by this patch.

Re: Bug#652359: notmuch-emacs won't display correctly quoted-printable iso-8859-1 mails

[Olivier Berger]

On Sun, 18 Dec 2011 17:55:47 -0400, David Bremner <david@tethera.net> wrote:
> On Sun, 18 Dec 2011 15:53:26 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> > I did that over notmuch-emacs 0.10.2-1 Debian package's version of
> > notmuch-query.el, but that doesn't seem change anything, unfortunately :
> > the modeline still is '-1:%*-' for the notmuch-show buffer, after
> > hitting RET over a message's line in a search result list :-(
> 
> Just to be clear, is the message displayed correctly in notmuch-show
> mode? 

No. Still UTF-8 characters displayed as strange glyphs instead of normal
french accentuated letters (classical problem of UTF-8 displayed in a
iso8859-1 mode).

> The modeline is not supposed to change, by this patch.
> 

If you say so ;) My wild guess : maybe that's the problem ? ... If it
did, then I'd expect the emacs rendering to be in sync between what it
is told to display and how it does it ? When displaying same emails with
Gnus, I can see accents correctly, and, guess what ? the modeline is indeed marking a
UTF-8 indicator instead of a latin1 one ;-)... but maybe I'm completely
getting it wrong.

Thanks for your time.

Best regards,
-- 
Olivier BERGER 
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)

Re: Bug#652359: notmuch-emacs: Won't display correctly quoted-printable iso-8859-1 mails

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Fri, 16 Dec 2011 17:01:04 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:

> In my understanding, the quoted-printable iso-8859-1 is correct for
> french accents, and is converted somehow to UTF-8, though it is
> displayed in an emacs buffer that looks like iso-8859-1 (The
> minibuffer separator line starts with '-1:%*-' ...

Do you the same behaviour with "emacs -q"?

I did the following: 

$ export LANG=fr_FR.utf8
$ export LC_CTYPE=fr_FR.utf8
$ emacs -q  # gtk emacs

M-x notmuch

and your message id:"877h1wv7mg.fsf@inf-8657.int-evry.fr" displays fine
(with U in the modeline).

Poking around at random, I did 

       M-x set-language-environment <Return> French <Return>

and then I could duplicate your problem; the mode line shows 8859-1
coding system and the message is not displayed correctly. On the other
hand, in this case the patch I sent does fix the actual message display
for me, just as it did in the other case where I set the locale directly
to fr_FR (without utf8).


d


[-- Attachment #2: mail-accentue --]
[-- Type: application/octet-stream, Size: 717 bytes --]

From: Olivier Berger <olivier.berger@it-sudparis.eu>
To: olivier.berger@it-sudparis.eu
Subject: Essai =?iso-8859-1?Q?accentu=E9?=
User-Agent: Notmuch/0.10.1 (http://notmuchmail.org) Emacs/23.3.1 (i486-pc-linux-gnu)
X-Draft-From: ("nnimap+localdovecot:INBOX" 44228)
Date: Fri, 16 Dec 2011 16:49:59 +0100
Message-ID: <877h1wv7mg.fsf@inf-8657.int-evry.fr>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Du texte accentu=E9 pour =E7a ...

=E0 la bonne heure !
--=20
Olivier BERGER=20
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)

Re: Bug#652359: notmuch-emacs: Won't display correctly quoted-printable iso-8859-1 mails

[Olivier Berger]

On Sun, 18 Dec 2011 20:03:43 -0400, David Bremner <david@tethera.net> wrote:
> On Fri, 16 Dec 2011 17:01:04 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> 
> > In my understanding, the quoted-printable iso-8859-1 is correct for
> > french accents, and is converted somehow to UTF-8, though it is
> > displayed in an emacs buffer that looks like iso-8859-1 (The
> > minibuffer separator line starts with '-1:%*-' ...
> 
> Do you the same behaviour with "emacs -q"?
> 

No.

> I did the following: 
> 
> $ export LANG=fr_FR.utf8
> $ export LC_CTYPE=fr_FR.utf8
> $ emacs -q  # gtk emacs
> 
> M-x notmuch
> 
> and your message id:"877h1wv7mg.fsf@inf-8657.int-evry.fr" displays fine
> (with U in the modeline).
> 

Same here :-) ... so there's something weird in my (emacs?)
environment...

> Poking around at random, I did 
> 
>        M-x set-language-environment <Return> French <Return>
> 
> and then I could duplicate your problem; the mode line shows 8859-1
> coding system and the message is not displayed correctly.

Same here. As well as with Latin-1.

On the contrary, UTF-8 sets it fine ;-) as well as if M-x
reset-language-environment, as expected ;)

> On the other hand, in this case the patch I sent does fix the actual
> message display for me, just as it did in the other case where I set
> the locale directly to fr_FR (without utf8).
> 

Strange : that doesn't suffice here... but there may be other
combinations of settings that differ from yours.

Anyway, we seem to have a workaround here, if I just reset the language environment...
But I wonder what other side-effects there may be.

Still, just for comparison, Gnus managed well... but maybe because it
doesn't work with UTF-8 ;-)

Thanks for your help.

Best regards,
-- 
Olivier BERGER 
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)

Re: Bug#652359: notmuch-emacs: Won't display correctly quoted-printable iso-8859-1 mails

[Olivier Berger]

Hi.

On Mon, 19 Dec 2011 11:25:04 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> 
> On the contrary, UTF-8 sets it fine ;-) as well as if M-x
> reset-language-environment, as expected ;)
> 

I've checked and had :
- current-language-environment "Latin-1"
- and default-input-method "latin-1-prefix"
in my customizations in .emacs.

I've removed these, and it seems much better now.

I don't remember why these we necessary initially... and hope I will be
able to live without these from now on.

Hope this helps.

Best regards,
-- 
Olivier BERGER 
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)

SOLVED - Was: Re: Bug#652359: notmuch-emacs: Won't display correctly quoted-printable iso-8859-1 mails

[Olivier Berger]

On Mon, 19 Dec 2011 11:34:29 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> Hi.
> 
> On Mon, 19 Dec 2011 11:25:04 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> > 
> > On the contrary, UTF-8 sets it fine ;-) as well as if M-x
> > reset-language-environment, as expected ;)
> > 
> 
> I've checked and had :
> - current-language-environment "Latin-1"
> - and default-input-method "latin-1-prefix"
> in my customizations in .emacs.
> 
> I've removed these, and it seems much better now.
> 
> I don't remember why these we necessary initially... and hope I will be
> able to live without these from now on.
> 

Just for completeness, it seems that it's the specific Latin-1 setting
of current-language-environment that messed with the rendering.

If I don't customize it to Latin-1, the current-language-environment
seems to be calculated to French (which sounds logical given my locale
envs), and then it works. I assume that French means French with UTF-8
whereas Latin-1 means what it means ;)

That's logical, but far from obvious until you've found the culprit.

Closing the Debian bug then.

Sorry for the bothering, and thanks for your help.

/Merci et à bientôt/ (just to check once more ;)

Best regards,
-- 
Olivier BERGER 
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingenieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)

Re: SOLVED - Was: Re: Bug#652359: notmuch-emacs: Won't display correctly quoted-printable iso-8859-1 mails

[Tomi Ollila]

On Mon, 19 Dec 2011 11:53:55 +0100, Olivier Berger <olivier.berger@it-sudparis.eu> wrote:
> 
> Just for completeness, it seems that it's the specific Latin-1 setting
> of current-language-environment that messed with the rendering.
> 
> If I don't customize it to Latin-1, the current-language-environment
> seems to be calculated to French (which sounds logical given my locale
> envs), and then it works. I assume that French means French with UTF-8
> whereas Latin-1 means what it means ;)
> 
> That's logical, but far from obvious until you've found the culprit.
> 
> Closing the Debian bug then.
> 
> Sorry for the bothering, and thanks for your help.

Thank you for your bug report; we discovered another bug (possibly two)
how notmuch emacs mua reads data from external processes it executes.

> /Merci et à bientôt/ (just to check once more ;)
> 
> Best regards,
> -- 
> Olivier BERGER 

Tomi