Why some characters are not displayed correctly when setting mm-text-html-renderer to links

Why some characters are not displayed correctly when setting mm-text-html-renderer to links

[Wenlong Dai]

[-- Attachment #1.1: Type: text/plain, Size: 1289 bytes --]

I'm using gccemcs under WSL2 and trying to switch from mu4e to notmuch.
One thing I immediately like about notmuch is that it's much more faster
than mu4e.

But I also almost immediately ran into some issues. The one that bothers me
the most is,
when I hit ENTER to open a long thread from notmuch-search mode, it takes a
very long time to load.
Sometimes, this can take 30 seconds or more. My understanding is that
notmuch needs to load all the messages
(in my case, I tried threads with 20+ messages), and the default shr
renderer is not great at dealing with large amount of data.

After some research, I found that setting mm-text-html-renderer to links
would reduce the load time from <a long time>
to around 2 seconds or less. But one small issue with this is some
characters are not displayed correctly, for example the
RIGHT SINGLE QUOTATION MARK, which would be displayed as \342\200\231

A bit more research lead me to the toggle-enable-multibyte-characters
function, I found that if I invoke this function
twice interactively, the display would be fixed.

I guess my question is, how do I make notmuch-show display characters like
RIGHT SINGLE QUOTATION MARK correctly
when I use links as html renderer? And why would invoking
 toggle-enable-multibyte-characters fix the issue?

[-- Attachment #1.2: Type: text/html, Size: 3240 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Why some characters are not displayed correctly when setting mm-text-html-renderer to links

[David Bremner]

Wenlong Dai <whudwl@gmail.com> writes:

>
> After some research, I found that setting mm-text-html-renderer to links
> would reduce the load time from <a long time>
> to around 2 seconds or less. But one small issue with this is some
> characters are not displayed correctly, for example the
> RIGHT SINGLE QUOTATION MARK, which would be displayed as \342\200\231
>

I didn't have time to debug this yet, but I did duplicate the problem on
Emacs 27.1 on Debian, so it is not Windows or gccemacs specific. As far
as I can tell links is outputing non-utf8 characters, but it could also
be other issues like double decoding.

d

Re: Why some characters are not displayed correctly when setting mm-text-html-renderer to links

[Tomi Ollila]

On Fri, Aug 06 2021, David Bremner wrote:

> Wenlong Dai <whudwl@gmail.com> writes:
>
>>
>> After some research, I found that setting mm-text-html-renderer to links
>> would reduce the load time from <a long time>
>> to around 2 seconds or less. But one small issue with this is some
>> characters are not displayed correctly, for example the
>> RIGHT SINGLE QUOTATION MARK, which would be displayed as \342\200\231
>>
>
> I didn't have time to debug this yet, but I did duplicate the problem on
> Emacs 27.1 on Debian, so it is not Windows or gccemacs specific. As far
> as I can tell links is outputing non-utf8 characters, but it could also
> be other issues like double decoding.

setting those "coding-systems" is somewhat hard to understand (or it is
just me) -- i recall wondering these

  (let ((coding-system-for-read 'no-conversion))

iirc in related code...

One thing could be to wrap the 'links' command in a shell script
which executes 

  exec strace -o /tmp/strace.$$ -e trace=open /path/to/links "$@"'

(name that script as 'links' and point first entry in PATH to the 
directory that exists)

then one can check the logs what is links outputting (i'd guess it
would output utf-8 characters but one cannot know for sure...)

Tomi

>
> d

Re: Why some characters are not displayed correctly when setting mm-text-html-renderer to links

[David Bremner]

David Bremner <david@tethera.net> writes:

> Wenlong Dai <whudwl@gmail.com> writes:
>
>>
>> After some research, I found that setting mm-text-html-renderer to links
>> would reduce the load time from <a long time>
>> to around 2 seconds or less. But one small issue with this is some
>> characters are not displayed correctly, for example the
>> RIGHT SINGLE QUOTATION MARK, which would be displayed as \342\200\231
>>
>
> I didn't have time to debug this yet, but I did duplicate the problem on
> Emacs 27.1 on Debian, so it is not Windows or gccemacs specific. As far
> as I can tell links is outputing non-utf8 characters, but it could also
> be other issues like double decoding.
>

Thanks to Lars Ingebrigtsen, the problem will be fixed in emacs 28, as
of upstream emaacs commit a2c75d349a3ab83.

d