bug#27141: 26.0.50; mml-generate-mime-1 broken

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

Hi,

I noticed this when I forwarded a mail using Gnus.  The problem
is that non-ASCII letters contained in the header of the original
mail will not be encoded properly.  The rough outline of what
`mml-generate-mime-1' does then are:

・Insert the original mail (decoded) into a unibyte buffer
  (that with-temp-buffer of mml.el-[line:619] generates).
・Detect the charset of the contents.
・Encode the contents by that detected charset.
・Insert the encoded contsnts into the other unibyte buffer
  (that with-temp-buffer of mml.el-[line:612] generates).
・Run `mml-to-mime' that encodes the header of the original mail
  using `mail-encode-encoded-word-buffer' (an alias to
  `rfc2047-encode-message-header').

rfc2047's encoder expects human readable text, however, there
are encoded ones as described above.  So, as for at least mail
forwarding, `mml-generate-mime-1' is seriously broken.

The current code looks like WIP to me.  How about reverting it
to the emacs-25 version?  Slightly modernized[1] one is attached.

[1] Don't use old compat functions and string-as-(multi|uni)byte.

In GNU Emacs 26.0.50 (build 1, i686-pc-cygwin, GTK+ Version 3.18.9)
 of 2017-05-30 built on localhost
Windowing system distributor 'The Cygwin/X Project', version 11.0.11900000

[-- Attachment #2: Type: application/emacs-lisp, Size: 9199 bytes --]

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Katsumi Yamaoka]

On Tue, 30 May 2017 15:09:28 +0900, Katsumi Yamaoka wrote:
> So, as for at least mail forwarding, `mml-generate-mime-1' is
> seriously broken.

I've reverted it to the emacs-25 version (with some modifications)
for a emergency fix.

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Lars Ingebrigtsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> On Tue, 30 May 2017 15:09:28 +0900, Katsumi Yamaoka wrote:
>> So, as for at least mail forwarding, `mml-generate-mime-1' is
>> seriously broken.
>
> I've reverted it to the emacs-25 version (with some modifications)
> for a emergency fix.

Well, the function was reworked because it destroys files when
forwarding them, I seem to recall?  So this seems like it's just
changing one bug for another...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Katsumi Yamaoka]

On Thu, 01 Jun 2017 07:20:13 +0200, Lars Ingebrigtsen wrote:
> Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> On Tue, 30 May 2017 15:09:28 +0900, Katsumi Yamaoka wrote:
>>> So, as for at least mail forwarding, `mml-generate-mime-1' is
>>> seriously broken.

>> I've reverted it to the emacs-25 version (with some modifications)
>> for a emergency fix.

> Well, the function was reworked because it destroys files when
> forwarding them, I seem to recall?  So this seems like it's just
> changing one bug for another...

OK, could you please re-revert and fix it?  I did have no idea
to fix the bug in this construction...

(with-temp-buffer
  (set-buffer-multibyte nil)
  (insert
   (with-temp-buffer ;; <- unibyte
     (insert something) ;; <- multibyte
     (encode-coding ...)
     (buffer-string))))

Thanks.

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Lars Ingebrigtsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> OK, could you please re-revert and fix it?  I did have no idea
> to fix the bug in this construction...
>
> (with-temp-buffer
>   (set-buffer-multibyte nil)
>   (insert
>    (with-temp-buffer ;; <- unibyte

Hm...  why is this buffer unibyte here?  And if it is, won't
(set-buffer-multibyte t) fix that?

>      (insert something) ;; <- multibyte
>      (encode-coding ...)
>      (buffer-string))))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Katsumi Yamaoka]

On Thu, 01 Jun 2017 09:17:06 +0200, Lars Ingebrigtsen wrote:
>> (with-temp-buffer
>>   (set-buffer-multibyte nil)
>>   (insert
>>    (with-temp-buffer ;; <- unibyte

> Hm...  why is this buffer unibyte here?

Oops, for years I misunderstood that the multibyteness of a newly
created buffer defaults to that of the current buffer.  But it's
wrong!

(with-temp-buffer
  (set-buffer-multibyte nil)
  (with-temp-buffer
    enable-multibyte-characters))
 => t

> And if it is, won't (set-buffer-multibyte t) fix that?

No, it doesn't help.  As I wrote the first post,

> ・Insert the original mail (decoded) into a unibyte buffer
                                              ^^^^^^^ multibyte
>   (that with-temp-buffer of mml.el-[line:619] generates).
> ・Detect the charset of the contents.
> ・Encode the contents by that detected charset.
> ・Insert the encoded contsnts into the other unibyte buffer
>   (that with-temp-buffer of mml.el-[line:612] generates).
> ・Run `mml-to-mime' that encodes the header of the original mail
>   using `mail-encode-encoded-word-buffer' (an alias to
>   `rfc2047-encode-message-header').

> rfc2047's encoder expects human readable text, however, there
> are encoded ones as described above.

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Eli Zaretskii]

> Date: Thu, 01 Jun 2017 17:32:53 +0900
> From: Katsumi Yamaoka <yamaoka@jpl.org>
> Cc: 27141@debbugs.gnu.org
> 
> On Thu, 01 Jun 2017 09:17:06 +0200, Lars Ingebrigtsen wrote:
> >> (with-temp-buffer
> >>   (set-buffer-multibyte nil)
> >>   (insert
> >>    (with-temp-buffer ;; <- unibyte
> 
> > Hm...  why is this buffer unibyte here?
> 
> Oops, for years I misunderstood that the multibyteness of a newly
> created buffer defaults to that of the current buffer.  But it's
> wrong!
> 
> (with-temp-buffer
>   (set-buffer-multibyte nil)
>   (with-temp-buffer
>     enable-multibyte-characters))
>  => t
> 
> > And if it is, won't (set-buffer-multibyte t) fix that?
> 
> No, it doesn't help.  As I wrote the first post,
> 
> > ・Insert the original mail (decoded) into a unibyte buffer
>                                               ^^^^^^^ multibyte
> >   (that with-temp-buffer of mml.el-[line:619] generates).
> > ・Detect the charset of the contents.
> > ・Encode the contents by that detected charset.
> > ・Insert the encoded contsnts into the other unibyte buffer
> >   (that with-temp-buffer of mml.el-[line:612] generates).
> > ・Run `mml-to-mime' that encodes the header of the original mail
> >   using `mail-encode-encoded-word-buffer' (an alias to
> >   `rfc2047-encode-message-header').
> 
> > rfc2047's encoder expects human readable text, however, there
> > are encoded ones as described above.

Ping!  Any news on this issue?  It currently indicated as blocking the
release of Emacs 26.1, so can we please expedite its resolution?

Thanks.

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Katsumi Yamaoka]

On Sat, 02 Sep 2017 17:05:05 +0300, Eli Zaretskii wrote:
> Ping!  Any news on this issue?  It currently indicated as blocking the
> release of Emacs 26.1, so can we please expedite its resolution?

The version Lars modified was obviously broken at least for the
problem I brought up:

> I noticed this when I forwarded a mail using Gnus.  The problem
> is that non-ASCII letters contained in the header of the original
> mail will not be encoded properly.

Oh, I forgot to write the bottom line, sorry.  It happend when I
forward a message in which the header contains non-ASCII letters.
What I did then (revno 8130d91) was reverting Lars' change.  I'm
using the current version with no problem so far.  Though I don't
know what Lars tried to fix, I think it is not a stopper for the
next release as there seems to be no bug report brought up about
it.  Lars?

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Lars Ingebrigtsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Oh, I forgot to write the bottom line, sorry.  It happend when I
> forward a message in which the header contains non-ASCII letters.
> What I did then (revno 8130d91) was reverting Lars' change.  I'm
> using the current version with no problem so far.  Though I don't
> know what Lars tried to fix, I think it is not a stopper for the
> next release as there seems to be no bug report brought up about
> it.  Lars?

I seem to have forgotten to tag my patch with the corresponding bug
report?  Annoying.  Anyway, the problem my patch fixed was dealing with
text files included in Message with mml tags -- they would (under some
circumstances; there was a test case in the original bug report) be
corrupted.

But my fix was obviously a mistake, since it resulted in pretty much all
multibyte buffers being mishandled when forwarding, so the original
problem will have to be fixed in a different way.

But I don't think this should be a release blocker.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Eli Zaretskii]

unblock 24655 by 27141
thanks

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  27141@debbugs.gnu.org
> Date: Wed, 13 Sep 2017 19:36:47 +0200
> 
> I seem to have forgotten to tag my patch with the corresponding bug
> report?  Annoying.  Anyway, the problem my patch fixed was dealing with
> text files included in Message with mml tags -- they would (under some
> circumstances; there was a test case in the original bug report) be
> corrupted.
> 
> But my fix was obviously a mistake, since it resulted in pretty much all
> multibyte buffers being mishandled when forwarding, so the original
> problem will have to be fixed in a different way.
> 
> But I don't think this should be a release blocker.

Thanks, I'm unblocking it.  Though I hope this will be fixed before
26.1 is out.

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, I'm unblocking it.  Though I hope this will be fixed before
> 26.1 is out.

Yup.  Does anybody happen to know what the original bug report number
is?  I seem to remember that it had a nice test case that demonstrated
the problem...  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yamaoka@jpl.org,  27141@debbugs.gnu.org
> Date: Wed, 13 Sep 2017 20:45:11 +0200
> 
> Does anybody happen to know what the original bug report number is?
> I seem to remember that it had a nice test case that demonstrated
> the problem...  :-)

Not sure what you mean.  Did you mean this:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27141#5

I see no other bug number cited in the discussion of this one.

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Not sure what you mean.  Did you mean this:
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27141#5
>
> I see no other bug number cited in the discussion of this one.

There is no other bug number mentioned, which is why I asked whether
someone (by chance) remembered the other bug number.  It was a long
shot.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27141: 26.0.50; mml-generate-mime-1 broken

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Not sure what you mean.  Did you mean this:
>>
>>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27141#5
>>
>> I see no other bug number cited in the discussion of this one.
>
> There is no other bug number mentioned, which is why I asked whether
> someone (by chance) remembered the other bug number.  It was a long
> shot.  :-)

I've looked around, and I can't find it.  Without a test case, I don't
think we're going to get further on that problem, and this bug report is
about the bug my fix introduced, so I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no