Why multibyte for original-article buffer

Why multibyte for original-article buffer

[Stefan Monnier]

I've been running with the following patch with good results.
Also the patch makes sense to me: the original article is a sequence of
bytes (i.e. encoded chars), so a unibyte buffer makes a lot of sense
(it's more efficient and is likely to hide fewer bugs).

Yet the code currently explicitly sets the buffer to multibyte mode.
Does anybody know why?


        Stefan


--- gnus-art.el.~1.154.~	2008-02-28 14:10:39.000000000 -0500
+++ gnus-art.el	2008-02-28 14:09:51.000000000 -0500
@@ -4342,7 +4342,7 @@
     (gnus-article-setup-highlight-words)
     ;; Init original article buffer.
     (with-current-buffer (gnus-get-buffer-create gnus-original-article-buffer)
-      (mm-enable-multibyte)
+      (mm-disable-multibyte)
       (setq major-mode 'gnus-original-article-mode)
       (make-local-variable 'gnus-original-article))
     (if (and (get-buffer name)

Re: Why multibyte for original-article buffer

[Katsumi Yamaoka]

>>>>> Stefan Monnier wrote:

> I've been running with the following patch with good results.
> Also the patch makes sense to me: the original article is a sequence of
> bytes (i.e. encoded chars), so a unibyte buffer makes a lot of sense
> (it's more efficient and is likely to hide fewer bugs).

So, it is better to make also the " *nntpd*" buffer, the process
buffer for pop3, etc. be unibyte.  If such changes cause a problem,
it will be due to a code that copies data from those buffers to
a multibyte buffer and then decodes the data.  Actually, making
the original article buffer be unibyte prevents me from reading
a Japanese 8bit shift_jis message like this.

日本語

(Try `g' in the summary buffer.)

> Yet the code currently explicitly sets the buffer to multibyte mode.
> Does anybody know why?

IIRC, it was done about ten years ago.  I guess it was probably
a workaround for a problem like the one mentioned above.

> --- gnus-art.el.~1.154.~	2008-02-28 14:10:39.000000000 -0500
> +++ gnus-art.el	2008-02-28 14:09:51.000000000 -0500
> @@ -4342,7 +4342,7 @@
>      (gnus-article-setup-highlight-words)
>      ;; Init original article buffer.
>      (with-current-buffer (gnus-get-buffer-create gnus-original-article-buffer)
> -      (mm-enable-multibyte)
> +      (mm-disable-multibyte)
>        (setq major-mode 'gnus-original-article-mode)
>        (make-local-variable 'gnus-original-article))
>      (if (and (get-buffer name)

It is beyond my capacity to verify (and possibly to fix) all the
Gnus codes that copy data from there to somewhere.

Regards,

Re: Why multibyte for original-article buffer

[Stefan Monnier]

>> I've been running with the following patch with good results.
>> Also the patch makes sense to me: the original article is a sequence of
>> bytes (i.e. encoded chars), so a unibyte buffer makes a lot of sense
>> (it's more efficient and is likely to hide fewer bugs).

> So, it is better to make also the " *nntpd*" buffer, the process
> buffer for pop3, etc. be unibyte.  If such changes cause a problem,
> it will be due to a code that copies data from those buffers to
> a multibyte buffer and then decodes the data.  Actually, making
> the original article buffer be unibyte prevents me from reading
> a Japanese 8bit shift_jis message like this.

> 日本語

> (Try `g' in the summary buffer.)

Indeed, I see that.  The patch below will fix it.

>> Yet the code currently explicitly sets the buffer to multibyte mode.
>> Does anybody know why?

> IIRC, it was done about ten years ago.  I guess it was probably
> a workaround for a problem like the one mentioned above.

Yes, the Gnus code is full of such things.  I suggest it's time to take
a step back and think about how things *should* work.  Then change the
parts for which we know how they should work.  This will introduce bugs,
but then we'll know that the bugs are elsewhere and that fixing them
will get us closer to a good solution.


        Stefan


@@ -6564,7 +6564,13 @@
 		 (with-current-buffer gnus-original-article-buffer
 		   (and (equal (car gnus-original-article) group)
 			(eq (cdr gnus-original-article) article))))
-	    (insert-buffer-substring gnus-original-article-buffer)
+            ;; `insert-buffer-substring' would incorrectly use the
+            ;; equivalent of string-make-multibyte which amounts to decoding
+            ;; with locale-coding-system, causing failure of
+            ;; subsequent decoding.
+            (insert (mm-string-to-multibyte
+                     (with-current-buffer gnus-original-article-buffer
+                       (buffer-substring (point-min) (point-max)))))
 	    'article)
 	   ;; Check the backlog.
 	   ((and gnus-keep-backlog

Re: Why multibyte for original-article buffer

[Katsumi Yamaoka]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Indeed, I see that.  The patch below will fix it.

It works.

> -	    (insert-buffer-substring gnus-original-article-buffer)
> +            ;; `insert-buffer-substring' would incorrectly use the
> +            ;; equivalent of string-make-multibyte which amounts to decoding
> +            ;; with locale-coding-system, causing failure of
> +            ;; subsequent decoding.
> +            (insert (mm-string-to-multibyte
> +                     (with-current-buffer gnus-original-article-buffer
> +                       (buffer-substring (point-min) (point-max)))))

This might be the most suitable occasion to decode raw data into
the human readable ones rather than only converting the multibyteness.
It suggests there will also be such items lying about here and
there.  Though I think it should be done for the second step.

> Yes, the Gnus code is full of such things.  I suggest it's time to take
> a step back and think about how things *should* work.  Then change the
> parts for which we know how they should work.  This will introduce bugs,
> but then we'll know that the bugs are elsewhere and that fixing them
> will get us closer to a good solution.

Great.  I agree with installing those two changes as the first
step, though it might force non-ASCII people to live hard with
No Gnus for a while,   I will one by one fix or at least report
things caused by it.

Regards,

Re: Why multibyte for original-article buffer

[Miles Bader]

Katsumi Yamaoka <yamaoka@jpl.org> writes:
> though it might force non-ASCII people to live hard with
> No Gnus for a while

That's their job... :-)

-Miles

-- 
Abstainer, n. A weak person who yields to the temptation of denying himself a
pleasure. A total abstainer is one who abstains from everything but
abstention, and especially from inactivity in the affairs of others.