Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Florian Weimer]

* Lars Ingebrigtsen:

> branch: master
> commit ef14acfb68bb5b0ce42221e9681b93562f8085eb
> Author: Lars Ingebrigtsen <larsi@gnus.org>
> Commit: Lars Ingebrigtsen <larsi@gnus.org>
>
>     Make nnml handle invalid non-ASCII headers more consistently
>     
>     * lisp/gnus/nnml.el (nnml--encode-headers): New function to
>     RFC2047-encode invalid Subject/From headers (bug#45925).  This
>     will make them be displayed more consistently in the Summary
>     buffer (but still "wrong" sometimes, since there's not that much
>     we can guess at at this stage, charset wise).
>     (nnml-parse-head): Use it.
> ---
>  lisp/gnus/nnml.el | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>
> diff --git a/lisp/gnus/nnml.el b/lisp/gnus/nnml.el
> index ebececa..3cdfc74 100644
> --- a/lisp/gnus/nnml.el
> +++ b/lisp/gnus/nnml.el
> @@ -769,8 +769,24 @@ article number.  This function is called narrowed to an article."
>        (let ((headers (nnheader-parse-head t)))
>  	(setf (mail-header-chars  headers) chars)
>  	(setf (mail-header-number headers) number)
> +	;; If there's non-ASCII raw characters in the data,
> +	;; RFC2047-encode them to avoid having arbitrary data in the
> +	;; .overview file.
> +	(nnml--encode-headers headers)
>  	headers))))

Unfortunately, this change in particular causes Gnus to stops storing
messages into nnmail after receiving a message with this header:

From: =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?=
 <elfutils-devel@sourceware.org>

The logged error message is:

Mail source (maildir :path …) failed: (error Invalid data for rfc2047 encoding: 罗勇刚(Yonggang Luo) via Elfutils-devel <elfutils-devel@sourceware.org>)

On an older Emacs without this change, it seems that the original header
is written to the .overview file, which sidestep the problem that not
all strings are encodable by the rfc2047 functions.

Thanks,
Florian

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Eli Zaretskii]

> From: Florian Weimer <fweimer@redhat.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, ding@gnus.org
> Date: Fri, 16 Dec 2022 23:42:21 +0100
> 
> * Lars Ingebrigtsen:
> 
> > branch: master
> > commit ef14acfb68bb5b0ce42221e9681b93562f8085eb
> > Author: Lars Ingebrigtsen <larsi@gnus.org>
> > Commit: Lars Ingebrigtsen <larsi@gnus.org>
> >
> >     Make nnml handle invalid non-ASCII headers more consistently
> >     
> >     * lisp/gnus/nnml.el (nnml--encode-headers): New function to
> >     RFC2047-encode invalid Subject/From headers (bug#45925).  This
> >     will make them be displayed more consistently in the Summary
> >     buffer (but still "wrong" sometimes, since there's not that much
> >     we can guess at at this stage, charset wise).
> >     (nnml-parse-head): Use it.
> > ---
> >  lisp/gnus/nnml.el | 16 ++++++++++++++++
> >  1 file changed, 16 insertions(+)
> >
> > diff --git a/lisp/gnus/nnml.el b/lisp/gnus/nnml.el
> > index ebececa..3cdfc74 100644
> > --- a/lisp/gnus/nnml.el
> > +++ b/lisp/gnus/nnml.el
> > @@ -769,8 +769,24 @@ article number.  This function is called narrowed to an article."
> >        (let ((headers (nnheader-parse-head t)))
> >  	(setf (mail-header-chars  headers) chars)
> >  	(setf (mail-header-number headers) number)
> > +	;; If there's non-ASCII raw characters in the data,
> > +	;; RFC2047-encode them to avoid having arbitrary data in the
> > +	;; .overview file.
> > +	(nnml--encode-headers headers)
> >  	headers))))
> 
> Unfortunately, this change in particular causes Gnus to stops storing
> messages into nnmail after receiving a message with this header:
> 
> From: =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?=
>  <elfutils-devel@sourceware.org>
> 
> The logged error message is:
> 
> Mail source (maildir :path …) failed: (error Invalid data for rfc2047 encoding: 罗勇刚(Yonggang Luo) via Elfutils-devel <elfutils-devel@sourceware.org>)
> 
> On an older Emacs without this change, it seems that the original header
> is written to the .overview file, which sidestep the problem that not
> all strings are encodable by the rfc2047 functions.

Thanks.  I guess this From header is invalid because there's no space
between the "罗勇刚" and the "(Yonggang Luo)" parts?

Does the naïve patch below solve the problem?

diff --git a/lisp/gnus/nnml.el b/lisp/gnus/nnml.el
index 40e4b9e..7aa445e 100644
--- a/lisp/gnus/nnml.el
+++ b/lisp/gnus/nnml.el
@@ -776,17 +776,22 @@ nnml-parse-head
 	(nnml--encode-headers headers)
 	headers))))
 
+;; RFC2047-encode Subject and From, but leave invalid headers unencoded.
 (defun nnml--encode-headers (headers)
   (let ((subject (mail-header-subject headers))
 	(rfc2047-encoding-type 'mime))
     (unless (string-match "\\`[[:ascii:]]*\\'" subject)
-      (setf (mail-header-subject headers)
-	    (mail-encode-encoded-word-string subject t))))
+      (let ((encoded-subject
+             (ignore-errors (mail-encode-encoded-word-string subject t))))
+        (if encoded-subject
+            (setf (mail-header-subject headers) encoded-subject)))))
   (let ((from (mail-header-from headers))
 	(rfc2047-encoding-type 'address-mime))
     (unless (string-match "\\`[[:ascii:]]*\\'" from)
-      (setf (mail-header-from headers)
-	    (rfc2047-encode-string from t)))))
+      (let ((encoded-from
+             (ignore-errors (rfc2047-encode-string from t))))
+        (if encoded-from
+            (setf (mail-header-from headers) encoded-from))))))
 
 (defun nnml-get-nov-buffer (group &optional incrementalp)
   (let ((buffer (gnus-get-buffer-create

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Florian Weimer]

* Eli Zaretskii:

>> From: Florian Weimer <fweimer@redhat.com>
>> Cc: Lars Ingebrigtsen <larsi@gnus.org>, ding@gnus.org
>> Date: Fri, 16 Dec 2022 23:42:21 +0100
>> 
>> * Lars Ingebrigtsen:
>> 
>> > branch: master
>> > commit ef14acfb68bb5b0ce42221e9681b93562f8085eb
>> > Author: Lars Ingebrigtsen <larsi@gnus.org>
>> > Commit: Lars Ingebrigtsen <larsi@gnus.org>
>> >
>> >     Make nnml handle invalid non-ASCII headers more consistently
>> >     
>> >     * lisp/gnus/nnml.el (nnml--encode-headers): New function to
>> >     RFC2047-encode invalid Subject/From headers (bug#45925).  This
>> >     will make them be displayed more consistently in the Summary
>> >     buffer (but still "wrong" sometimes, since there's not that much
>> >     we can guess at at this stage, charset wise).
>> >     (nnml-parse-head): Use it.
>> > ---
>> >  lisp/gnus/nnml.el | 16 ++++++++++++++++
>> >  1 file changed, 16 insertions(+)
>> >
>> > diff --git a/lisp/gnus/nnml.el b/lisp/gnus/nnml.el
>> > index ebececa..3cdfc74 100644
>> > --- a/lisp/gnus/nnml.el
>> > +++ b/lisp/gnus/nnml.el
>> > @@ -769,8 +769,24 @@ article number.  This function is called narrowed to an article."
>> >        (let ((headers (nnheader-parse-head t)))
>> >  	(setf (mail-header-chars  headers) chars)
>> >  	(setf (mail-header-number headers) number)
>> > +	;; If there's non-ASCII raw characters in the data,
>> > +	;; RFC2047-encode them to avoid having arbitrary data in the
>> > +	;; .overview file.
>> > +	(nnml--encode-headers headers)
>> >  	headers))))
>> 
>> Unfortunately, this change in particular causes Gnus to stops storing
>> messages into nnmail after receiving a message with this header:
>> 
>> From: =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?=
>>  <elfutils-devel@sourceware.org>
>> 
>> The logged error message is:
>> 
>> Mail source (maildir :path …) failed: (error Invalid data for rfc2047 encoding: 罗勇刚(Yonggang Luo) via Elfutils-devel <elfutils-devel@sourceware.org>)
>> 
>> On an older Emacs without this change, it seems that the original header
>> is written to the .overview file, which sidestep the problem that not
>> all strings are encodable by the rfc2047 functions.
>
> Thanks.  I guess this From header is invalid because there's no space
> between the "罗勇刚" and the "(Yonggang Luo)" parts?

Yes, that seems to be what's tripping the encoder.  But I'm not sure if
proper encoding of ( or ) (as =28 or =29 using the Q encoding, or using
the B encoding as in the raw text) is actually invalid.  RFC 2047 only
talks about unencoded ( or ).  In contrast, encoded ( and ) are valid
syntax at the RFC 822 layer because encoding hides them.

> Does the naïve patch below solve the problem?
>
> diff --git a/lisp/gnus/nnml.el b/lisp/gnus/nnml.el
> index 40e4b9e..7aa445e 100644
> --- a/lisp/gnus/nnml.el
> +++ b/lisp/gnus/nnml.el
> @@ -776,17 +776,22 @@ nnml-parse-head
>  	(nnml--encode-headers headers)
>  	headers))))
>  
> +;; RFC2047-encode Subject and From, but leave invalid headers unencoded.
>  (defun nnml--encode-headers (headers)
>    (let ((subject (mail-header-subject headers))
>  	(rfc2047-encoding-type 'mime))
>      (unless (string-match "\\`[[:ascii:]]*\\'" subject)
> -      (setf (mail-header-subject headers)
> -	    (mail-encode-encoded-word-string subject t))))
> +      (let ((encoded-subject
> +             (ignore-errors (mail-encode-encoded-word-string subject t))))
> +        (if encoded-subject
> +            (setf (mail-header-subject headers) encoded-subject)))))
>    (let ((from (mail-header-from headers))
>  	(rfc2047-encoding-type 'address-mime))
>      (unless (string-match "\\`[[:ascii:]]*\\'" from)
> -      (setf (mail-header-from headers)
> -	    (rfc2047-encode-string from t)))))
> +      (let ((encoded-from
> +             (ignore-errors (rfc2047-encode-string from t))))
> +        (if encoded-from
> +            (setf (mail-header-from headers) encoded-from))))))
>  
>  (defun nnml-get-nov-buffer (group &optional incrementalp)
>    (let ((buffer (gnus-get-buffer-create

Thanks!

I somehow can't reproduce the original issue.  I expect more problematic
messages to arrive next week, though, and will report then how it goes.

Florian

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Eli Zaretskii]

> From: Florian Weimer <fweimer@redhat.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  Eric Abrahamsen
>  <eric@ericabrahamsen.net>,  emacs-devel@gnu.org,  ding@gnus.org
> Date: Sat, 17 Dec 2022 15:57:18 +0100
> 
> >> Mail source (maildir :path …) failed: (error Invalid data for rfc2047 encoding: 罗勇刚(Yonggang Luo) via Elfutils-devel <elfutils-devel@sourceware.org>)
> >> 
> >> On an older Emacs without this change, it seems that the original header
> >> is written to the .overview file, which sidestep the problem that not
> >> all strings are encodable by the rfc2047 functions.
> >
> > Thanks.  I guess this From header is invalid because there's no space
> > between the "罗勇刚" and the "(Yonggang Luo)" parts?
> 
> Yes, that seems to be what's tripping the encoder.  But I'm not sure if
> proper encoding of ( or ) (as =28 or =29 using the Q encoding, or using
> the B encoding as in the raw text) is actually invalid.  RFC 2047 only
> talks about unencoded ( or ).  In contrast, encoded ( and ) are valid
> syntax at the RFC 822 layer because encoding hides them.

I'm not sure this is related.  I was talking about the missing SPC
character before the parentheses.  But I know very little about this
stuff, so maybe I'm just misunderstanding you.

> > Does the naïve patch below solve the problem?
> >
> > diff --git a/lisp/gnus/nnml.el b/lisp/gnus/nnml.el
> > index 40e4b9e..7aa445e 100644
> > --- a/lisp/gnus/nnml.el
> > +++ b/lisp/gnus/nnml.el
> > @@ -776,17 +776,22 @@ nnml-parse-head
> >  	(nnml--encode-headers headers)
> >  	headers))))
> >  
> > +;; RFC2047-encode Subject and From, but leave invalid headers unencoded.
> >  (defun nnml--encode-headers (headers)
> >    (let ((subject (mail-header-subject headers))
> >  	(rfc2047-encoding-type 'mime))
> >      (unless (string-match "\\`[[:ascii:]]*\\'" subject)
> > -      (setf (mail-header-subject headers)
> > -	    (mail-encode-encoded-word-string subject t))))
> > +      (let ((encoded-subject
> > +             (ignore-errors (mail-encode-encoded-word-string subject t))))
> > +        (if encoded-subject
> > +            (setf (mail-header-subject headers) encoded-subject)))))
> >    (let ((from (mail-header-from headers))
> >  	(rfc2047-encoding-type 'address-mime))
> >      (unless (string-match "\\`[[:ascii:]]*\\'" from)
> > -      (setf (mail-header-from headers)
> > -	    (rfc2047-encode-string from t)))))
> > +      (let ((encoded-from
> > +             (ignore-errors (rfc2047-encode-string from t))))
> > +        (if encoded-from
> > +            (setf (mail-header-from headers) encoded-from))))))
> >  
> >  (defun nnml-get-nov-buffer (group &optional incrementalp)
> >    (let ((buffer (gnus-get-buffer-create
> 
> Thanks!
> 
> I somehow can't reproduce the original issue.  I expect more problematic
> messages to arrive next week, though, and will report then how it goes.

Thanks.  Please do get back, even if you don't succeed in reproducing.
I tested the code by invoking the body of nnml--encode-headers
directly on the header you provided, so I'm quite sure the fix will
prevent the error and will restore Gnus to how it worked before this
change, when such problematic headers are used.  So I'd like to
install this fix on the release branch for the upcoming Emacs 29.1,
and need your feedback.

TIA

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Florian Weimer]

* Eli Zaretskii:

>> From: Florian Weimer <fweimer@redhat.com>
>> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  Eric Abrahamsen
>>  <eric@ericabrahamsen.net>,  emacs-devel@gnu.org,  ding@gnus.org
>> Date: Sat, 17 Dec 2022 15:57:18 +0100
>> 
>> >> Mail source (maildir :path …) failed: (error Invalid data for rfc2047 encoding: 罗勇刚(Yonggang Luo) via Elfutils-devel <elfutils-devel@sourceware.org>)
>> >> 
>> >> On an older Emacs without this change, it seems that the original header
>> >> is written to the .overview file, which sidestep the problem that not
>> >> all strings are encodable by the rfc2047 functions.
>> >
>> > Thanks.  I guess this From header is invalid because there's no space
>> > between the "罗勇刚" and the "(Yonggang Luo)" parts?
>> 
>> Yes, that seems to be what's tripping the encoder.  But I'm not sure if
>> proper encoding of ( or ) (as =28 or =29 using the Q encoding, or using
>> the B encoding as in the raw text) is actually invalid.  RFC 2047 only
>> talks about unencoded ( or ).  In contrast, encoded ( and ) are valid
>> syntax at the RFC 822 layer because encoding hides them.
>
> I'm not sure this is related.  I was talking about the missing SPC
> character before the parentheses.  But I know very little about this
> stuff, so maybe I'm just misunderstanding you.

I was responding to “this From header is invalid”.  I'm not sure if the
original message is actually malformed.

>> I somehow can't reproduce the original issue.  I expect more problematic
>> messages to arrive next week, though, and will report then how it goes.
>
> Thanks.  Please do get back, even if you don't succeed in reproducing.

As I expected, I could reproduce the issue with new mail.  I'm happy to
report that replacing nnml--encode-headers with your new version fixes
the issue.

Going back to older mail, I see that previously the From: header was
written in encoded form to .overview (slightly wrapped to avoid SMTP
issues):

8523 Re: [PATCH 1/7] Rename 'hello2.spec.' -> 'hello2.spec' 'hello3.spec.' -> 'hello3.spec' =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?= <elfutils-devel@sourceware.org> Thu, 24 Nov 2022 16:27:56 +0800 <CAE2XoE_RBNdsPZMLiSQKJroyx4ycvqf7y+VDbuY=-XbhOwsL8g@mail.gmail.com>
  <20220920084307.1696-1-luoyonggang@gmail.com> <20220920084307.1696-2-luoyonggang@gmail.com> <Y0nH98k6YumMRSz0@wildebeest.org> <CAE2XoE-xfFQNMN1JYSi8Kp+-DUUookRkzwUV+OvKGfnZKDXKEA@mail.gmail.com> <20221019194930.GC24703@redhat.com> <3bf19d05c8976411432709fae1cc2bcc2d21d700.camel@klomp.org>
  <586fbfd512763e539f0d06e8ffa308da34be5368.camel@wildebeest.org> 705 32 Xref: oldenburg.str.redhat.com lists.sourceware.elfutils-devel:8523 Cc: "Frank Ch. Eigler" <fche@redhat.com>, elfutils-devel@sourceware.org To: Mark Wielaard <aegon@wildebeest.org>

Now it uses unencoded form:

8762 Re: [PATCH v2 13/16] lib: isatty is not available on windows 罗勇刚(Yonggang Luo) via Elfutils-devel <elfutils-devel@sourceware.org> Mon, 19 Dec 2022 20:32:48 +0800 <CAE2XoE-Xv6+4Npj38ih_nPMZ8e6ph7oS8wxvKFx1uE-sRfH_kw@mail.gmail.com> <20221217165213.152-1-luoyonggang@gmail.com>
  <20221217165213.152-14-luoyonggang@gmail.com> 1074 43 Xref: oldenburg.str.redhat.com lists.sourceware.elfutils-devel:8762 To: elfutils-devel@sourceware.org

I assume that there is some protection in place that tabs are not
possible here?

Thanks,
Florian

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Eli Zaretskii]

> From: Florian Weimer <fweimer@redhat.com>
> Cc: larsi@gnus.org,  eric@ericabrahamsen.net,  emacs-devel@gnu.org,
>   ding@gnus.org
> Date: Mon, 19 Dec 2022 14:59:17 +0100
> 
> > I'm not sure this is related.  I was talking about the missing SPC
> > character before the parentheses.  But I know very little about this
> > stuff, so maybe I'm just misunderstanding you.
> 
> I was responding to “this From header is invalid”.  I'm not sure if the
> original message is actually malformed.

Ah, okay.  I'm not sure, either, although I don't think I've ever seen
such addresses.  But I'm no expert.  Maybe rfc2047.el needs to be more
tolerant and less dependent on whitespace; I'll leave that to experts.
But even if there's nothing wrong with this particular header, I think
you will agree that signaling and error and thus failing the entire
storage of the message is not TRT, especially since the change which
causes this was done for basically aesthetic reasons.  Falling back to
previous behavior in these cases is better, I think.

> As I expected, I could reproduce the issue with new mail.  I'm happy to
> report that replacing nnml--encode-headers with your new version fixes
> the issue.

Thanks, so I've now installed this on the emacs-29 branch, for the
upcoming Emacs 29.1.

> Going back to older mail, I see that previously the From: header was
> written in encoded form to .overview (slightly wrapped to avoid SMTP
> issues):
> 
> 8523 Re: [PATCH 1/7] Rename 'hello2.spec.' -> 'hello2.spec' 'hello3.spec.' -> 'hello3.spec' =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?= <elfutils-devel@sourceware.org> Thu, 24 Nov 2022 16:27:56 +0800 <CAE2XoE_RBNdsPZMLiSQKJroyx4ycvqf7y+VDbuY=-XbhOwsL8g@mail.gmail.com>
>   <20220920084307.1696-1-luoyonggang@gmail.com> <20220920084307.1696-2-luoyonggang@gmail.com> <Y0nH98k6YumMRSz0@wildebeest.org> <CAE2XoE-xfFQNMN1JYSi8Kp+-DUUookRkzwUV+OvKGfnZKDXKEA@mail.gmail.com> <20221019194930.GC24703@redhat.com> <3bf19d05c8976411432709fae1cc2bcc2d21d700.camel@klomp.org>
>   <586fbfd512763e539f0d06e8ffa308da34be5368.camel@wildebeest.org> 705 32 Xref: oldenburg.str.redhat.com lists.sourceware.elfutils-devel:8523 Cc: "Frank Ch. Eigler" <fche@redhat.com>, elfutils-devel@sourceware.org To: Mark Wielaard <aegon@wildebeest.org>
> 
> Now it uses unencoded form:
> 
> 8762 Re: [PATCH v2 13/16] lib: isatty is not available on windows 罗勇刚(Yonggang Luo) via Elfutils-devel <elfutils-devel@sourceware.org> Mon, 19 Dec 2022 20:32:48 +0800 <CAE2XoE-Xv6+4Npj38ih_nPMZ8e6ph7oS8wxvKFx1uE-sRfH_kw@mail.gmail.com> <20221217165213.152-1-luoyonggang@gmail.com>
>   <20221217165213.152-14-luoyonggang@gmail.com> 1074 43 Xref: oldenburg.str.redhat.com lists.sourceware.elfutils-devel:8762 To: elfutils-devel@sourceware.org

Is that a problem?  I think this was part of the reason for that
change in nnml.el.

> I assume that there is some protection in place that tabs are not
> possible here?

Tabs where, and what kind of protection are we talking about?

Again, I'm not an expert on this and don't use Gnus, so apologies for
possibly dumb questions.

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Florian Weimer]

* Eli Zaretskii:

>> As I expected, I could reproduce the issue with new mail.  I'm happy to
>> report that replacing nnml--encode-headers with your new version fixes
>> the issue.
>
> Thanks, so I've now installed this on the emacs-29 branch, for the
> upcoming Emacs 29.1.

It's a denial-of-service issue, so it should be backported to 28 if
possible, too.

>> Going back to older mail, I see that previously the From: header was
>> written in encoded form to .overview (slightly wrapped to avoid SMTP
>> issues):
>> 
>> 8523 Re: [PATCH 1/7] Rename 'hello2.spec.' -> 'hello2.spec' 'hello3.spec.' -> 'hello3.spec' =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?= <elfutils-devel@sourceware.org> Thu, 24 Nov 2022 16:27:56 +0800 <CAE2XoE_RBNdsPZMLiSQKJroyx4ycvqf7y+VDbuY=-XbhOwsL8g@mail.gmail.com>
>>   <20220920084307.1696-1-luoyonggang@gmail.com> <20220920084307.1696-2-luoyonggang@gmail.com> <Y0nH98k6YumMRSz0@wildebeest.org> <CAE2XoE-xfFQNMN1JYSi8Kp+-DUUookRkzwUV+OvKGfnZKDXKEA@mail.gmail.com> <20221019194930.GC24703@redhat.com> <3bf19d05c8976411432709fae1cc2bcc2d21d700.camel@klomp.org>
>>   <586fbfd512763e539f0d06e8ffa308da34be5368.camel@wildebeest.org> 705 32 Xref: oldenburg.str.redhat.com lists.sourceware.elfutils-devel:8523 Cc: "Frank Ch. Eigler" <fche@redhat.com>, elfutils-devel@sourceware.org To: Mark Wielaard <aegon@wildebeest.org>
>> 
>> Now it uses unencoded form:
>> 
>> 8762 Re: [PATCH v2 13/16] lib: isatty is not available on windows 罗勇刚(Yonggang Luo) via Elfutils-devel <elfutils-devel@sourceware.org> Mon, 19 Dec 2022 20:32:48 +0800 <CAE2XoE-Xv6+4Npj38ih_nPMZ8e6ph7oS8wxvKFx1uE-sRfH_kw@mail.gmail.com> <20221217165213.152-1-luoyonggang@gmail.com>
>>   <20221217165213.152-14-luoyonggang@gmail.com> 1074 43 Xref: oldenburg.str.redhat.com lists.sourceware.elfutils-devel:8762 To: elfutils-devel@sourceware.org
>
> Is that a problem?  I think this was part of the reason for that
> change in nnml.el.

I don't know.

>> I assume that there is some protection in place that tabs are not
>> possible here?
>
> Tabs where, and what kind of protection are we talking about?

The .overview file is in tab-separated-values format, and I think the
code is supposed to replace tabs with spaces before the lines are
written.  But mail-encode-encoded-word-string/rfc2047-encode-string
generally does not replace tabs, so your change shouldn't make things
worse at least.

Thanks,
Florian

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Eli Zaretskii]

> From: Florian Weimer <fweimer@redhat.com>
> Cc: larsi@gnus.org,  eric@ericabrahamsen.net,  emacs-devel@gnu.org,
>   ding@gnus.org
> Date: Mon, 19 Dec 2022 18:27:09 +0100
> 
> * Eli Zaretskii:
> 
> >> As I expected, I could reproduce the issue with new mail.  I'm happy to
> >> report that replacing nnml--encode-headers with your new version fixes
> >> the issue.
> >
> > Thanks, so I've now installed this on the emacs-29 branch, for the
> > upcoming Emacs 29.1.
> 
> It's a denial-of-service issue, so it should be backported to 28 if
> possible, too.

We don't plan on any more releases from the emacs-28 branch.  I guess
distros will do the backporting, though.

> >> I assume that there is some protection in place that tabs are not
> >> possible here?
> >
> > Tabs where, and what kind of protection are we talking about?
> 
> The .overview file is in tab-separated-values format, and I think the
> code is supposed to replace tabs with spaces before the lines are
> written.  But mail-encode-encoded-word-string/rfc2047-encode-string
> generally does not replace tabs, so your change shouldn't make things
> worse at least.

OK, thanks.

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Eric Abrahamsen]

On 12/19/22 19:07 PM, Eli Zaretskii wrote:
>> From: Florian Weimer <fweimer@redhat.com>
>> Cc: larsi@gnus.org,  eric@ericabrahamsen.net,  emacs-devel@gnu.org,
>>   ding@gnus.org
>> Date: Mon, 19 Dec 2022 14:59:17 +0100
>> 
>> > I'm not sure this is related.  I was talking about the missing SPC
>> > character before the parentheses.  But I know very little about this
>> > stuff, so maybe I'm just misunderstanding you.
>> 
>> I was responding to “this From header is invalid”.  I'm not sure if the
>> original message is actually malformed.
>
> Ah, okay.  I'm not sure, either, although I don't think I've ever seen
> such addresses.  But I'm no expert.  Maybe rfc2047.el needs to be more
> tolerant and less dependent on whitespace; I'll leave that to experts.
> But even if there's nothing wrong with this particular header, I think
> you will agree that signaling and error and thus failing the entire
> storage of the message is not TRT, especially since the change which
> causes this was done for basically aesthetic reasons.  Falling back to
> previous behavior in these cases is better, I think.
>
>> As I expected, I could reproduce the issue with new mail.  I'm happy to
>> report that replacing nnml--encode-headers with your new version fixes
>> the issue.
>
> Thanks, so I've now installed this on the emacs-29 branch, for the
> upcoming Emacs 29.1.
>
>> Going back to older mail, I see that previously the From: header was
>> written in encoded form to .overview (slightly wrapped to avoid SMTP
>> issues):
>> 
>> 8523 Re: [PATCH 1/7] Rename 'hello2.spec.' -> 'hello2.spec'
>> 'hello3.spec.' -> 'hello3.spec'
>> =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?=
>> <elfutils-devel@sourceware.org> Thu, 24 Nov 2022 16:27:56 +0800
>> <CAE2XoE_RBNdsPZMLiSQKJroyx4ycvqf7y+VDbuY=-XbhOwsL8g@mail.gmail.com>
>>   <20220920084307.1696-1-luoyonggang@gmail.com>
>> <20220920084307.1696-2-luoyonggang@gmail.com>
>> <Y0nH98k6YumMRSz0@wildebeest.org>
>> <CAE2XoE-xfFQNMN1JYSi8Kp+-DUUookRkzwUV+OvKGfnZKDXKEA@mail.gmail.com>
>> <20221019194930.GC24703@redhat.com>
>> <3bf19d05c8976411432709fae1cc2bcc2d21d700.camel@klomp.org>
>>   <586fbfd512763e539f0d06e8ffa308da34be5368.camel@wildebeest.org>
>> 705 32 Xref: oldenburg.str.redhat.com
>> lists.sourceware.elfutils-devel:8523 Cc: "Frank Ch. Eigler"
>> <fche@redhat.com>, elfutils-devel@sourceware.org To: Mark Wielaard
>> <aegon@wildebeest.org>
>> 
>> Now it uses unencoded form:
>> 
>> 8762 Re: [PATCH v2 13/16] lib: isatty is not available on windows 罗
>> 勇刚(Yonggang Luo) via Elfutils-devel
>> <elfutils-devel@sourceware.org> Mon, 19 Dec 2022 20:32:48 +0800
>> <CAE2XoE-Xv6+4Npj38ih_nPMZ8e6ph7oS8wxvKFx1uE-sRfH_kw@mail.gmail.com>
>> <20221217165213.152-1-luoyonggang@gmail.com>
>>   <20221217165213.152-14-luoyonggang@gmail.com> 1074 43 Xref:
>> oldenburg.str.redhat.com lists.sourceware.elfutils-devel:8762 To:
>> elfutils-devel@sourceware.org
>
> Is that a problem?  I think this was part of the reason for that
> change in nnml.el.

It would definitely be a problem to change the on-disk representation of
data in the .overview files! Lots of code depends on a very specific
format for those files; I've broken things badly in the past by removing
header encoding there.

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Cc: Florian Weimer <fweimer@redhat.com>,  larsi@gnus.org,
>   emacs-devel@gnu.org,  ding@gnus.org
> Date: Mon, 19 Dec 2022 09:48:15 -0800
> 
> On 12/19/22 19:07 PM, Eli Zaretskii wrote:
> >> Going back to older mail, I see that previously the From: header was
> >> written in encoded form to .overview (slightly wrapped to avoid SMTP
> >> issues):
> >> 
> >> 8523 Re: [PATCH 1/7] Rename 'hello2.spec.' -> 'hello2.spec'
> >> 'hello3.spec.' -> 'hello3.spec'
> >> =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?=
> >> <elfutils-devel@sourceware.org> Thu, 24 Nov 2022 16:27:56 +0800
> >> <CAE2XoE_RBNdsPZMLiSQKJroyx4ycvqf7y+VDbuY=-XbhOwsL8g@mail.gmail.com>
> >>   <20220920084307.1696-1-luoyonggang@gmail.com>
> >> <20220920084307.1696-2-luoyonggang@gmail.com>
> >> <Y0nH98k6YumMRSz0@wildebeest.org>
> >> <CAE2XoE-xfFQNMN1JYSi8Kp+-DUUookRkzwUV+OvKGfnZKDXKEA@mail.gmail.com>
> >> <20221019194930.GC24703@redhat.com>
> >> <3bf19d05c8976411432709fae1cc2bcc2d21d700.camel@klomp.org>
> >>   <586fbfd512763e539f0d06e8ffa308da34be5368.camel@wildebeest.org>
> >> 705 32 Xref: oldenburg.str.redhat.com
> >> lists.sourceware.elfutils-devel:8523 Cc: "Frank Ch. Eigler"
> >> <fche@redhat.com>, elfutils-devel@sourceware.org To: Mark Wielaard
> >> <aegon@wildebeest.org>
> >> 
> >> Now it uses unencoded form:
> >> 
> >> 8762 Re: [PATCH v2 13/16] lib: isatty is not available on windows 罗
> >> 勇刚(Yonggang Luo) via Elfutils-devel
> >> <elfutils-devel@sourceware.org> Mon, 19 Dec 2022 20:32:48 +0800
> >> <CAE2XoE-Xv6+4Npj38ih_nPMZ8e6ph7oS8wxvKFx1uE-sRfH_kw@mail.gmail.com>
> >> <20221217165213.152-1-luoyonggang@gmail.com>
> >>   <20221217165213.152-14-luoyonggang@gmail.com> 1074 43 Xref:
> >> oldenburg.str.redhat.com lists.sourceware.elfutils-devel:8762 To:
> >> elfutils-devel@sourceware.org
> >
> > Is that a problem?  I think this was part of the reason for that
> > change in nnml.el.
> 
> It would definitely be a problem to change the on-disk representation of
> data in the .overview files! Lots of code depends on a very specific
> format for those files; I've broken things badly in the past by removing
> header encoding there.

But this change is there since Emacs 28, so if it's a problem, someone
would have hollered already, no?

Or are you saying that the changes I installed are causing the
difference?  If so, I don't see how that could happen, since the
change just ignores any errors signaled by rfc2047, and if rfc2047
does signal an error, what is stored in the file is the same string as
what we did before the new function nnml--encode-headers was added and
we started using it.

So unless I'm missing something, the changes I installed cannot cause
any new problems we didn't have since quite long ago.

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Cc: Florian Weimer <fweimer@redhat.com>,  larsi@gnus.org,
>>   emacs-devel@gnu.org,  ding@gnus.org
>> Date: Mon, 19 Dec 2022 09:48:15 -0800
>> 
>> On 12/19/22 19:07 PM, Eli Zaretskii wrote:
>> >> Going back to older mail, I see that previously the From: header was
>> >> written in encoded form to .overview (slightly wrapped to avoid SMTP
>> >> issues):
>> >> 
>> >> 8523 Re: [PATCH 1/7] Rename 'hello2.spec.' -> 'hello2.spec'
>> >> 'hello3.spec.' -> 'hello3.spec'
>> >> =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?=
>> >> <elfutils-devel@sourceware.org> Thu, 24 Nov 2022 16:27:56 +0800
>> >> <CAE2XoE_RBNdsPZMLiSQKJroyx4ycvqf7y+VDbuY=-XbhOwsL8g@mail.gmail.com>
>> >>   <20220920084307.1696-1-luoyonggang@gmail.com>
>> >> <20220920084307.1696-2-luoyonggang@gmail.com>
>> >> <Y0nH98k6YumMRSz0@wildebeest.org>
>> >> <CAE2XoE-xfFQNMN1JYSi8Kp+-DUUookRkzwUV+OvKGfnZKDXKEA@mail.gmail.com>
>> >> <20221019194930.GC24703@redhat.com>
>> >> <3bf19d05c8976411432709fae1cc2bcc2d21d700.camel@klomp.org>
>> >>   <586fbfd512763e539f0d06e8ffa308da34be5368.camel@wildebeest.org>
>> >> 705 32 Xref: oldenburg.str.redhat.com
>> >> lists.sourceware.elfutils-devel:8523 Cc: "Frank Ch. Eigler"
>> >> <fche@redhat.com>, elfutils-devel@sourceware.org To: Mark Wielaard
>> >> <aegon@wildebeest.org>
>> >> 
>> >> Now it uses unencoded form:
>> >> 
>> >> 8762 Re: [PATCH v2 13/16] lib: isatty is not available on windows 罗
>> >> 勇刚(Yonggang Luo) via Elfutils-devel
>> >> <elfutils-devel@sourceware.org> Mon, 19 Dec 2022 20:32:48 +0800
>> >> <CAE2XoE-Xv6+4Npj38ih_nPMZ8e6ph7oS8wxvKFx1uE-sRfH_kw@mail.gmail.com>
>> >> <20221217165213.152-1-luoyonggang@gmail.com>
>> >>   <20221217165213.152-14-luoyonggang@gmail.com> 1074 43 Xref:
>> >> oldenburg.str.redhat.com lists.sourceware.elfutils-devel:8762 To:
>> >> elfutils-devel@sourceware.org
>> >
>> > Is that a problem?  I think this was part of the reason for that
>> > change in nnml.el.
>> 
>> It would definitely be a problem to change the on-disk representation of
>> data in the .overview files! Lots of code depends on a very specific
>> format for those files; I've broken things badly in the past by removing
>> header encoding there.
>
> But this change is there since Emacs 28, so if it's a problem, someone
> would have hollered already, no?
>
> Or are you saying that the changes I installed are causing the
> difference?  If so, I don't see how that could happen, since the
> change just ignores any errors signaled by rfc2047, and if rfc2047
> does signal an error, what is stored in the file is the same string as
> what we did before the new function nnml--encode-headers was added and
> we started using it.
>
> So unless I'm missing something, the changes I installed cannot cause
> any new problems we didn't have since quite long ago.

Sorry, I misunderstood -- I thought this change was introducing a
difference in the saved files. My mistake for not following along
closely enough.

Maybe all this warrants some regression tests in ./test/lisp/mail?

Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Cc: fweimer@redhat.com,  larsi@gnus.org,  emacs-devel@gnu.org,  ding@gnus.org
> Date: Mon, 19 Dec 2022 10:05:24 -0800
> 
> Sorry, I misunderstood -- I thought this change was introducing a
> difference in the saved files. My mistake for not following along
> closely enough.

No sweat.

> Maybe all this warrants some regression tests in ./test/lisp/mail?

Adding tests is always welcome, TIA.