Gnus does not handle quoting in quoted strings correctly.

Gnus does not handle quoting in quoted strings correctly.

[Sascha Wilde]

Hi *,

if you have a mail with

From: "foo \[bar\]" <foo@bar.invalid>

in the headers, gnus displays the from literately as

foo \[bar\]

but according to RfC822/2822 "\" is used in quoted strings as special
quoting character and is always (no matter if necessary or not)
semantically "invisible".  So it should read:

foo [bar]


The other way around, when writing display names in quoted strings, no
quoting is done even when it would be needed.  Writing a message with

From: "foo/bar" <foo@bar.invalid>

the From: header is copied verbatim to the outgoing mail, but should
get quoted like:

From: "foo//bar" <foo@bar.invalid>

cheers
sascha
-- 
Sascha Wilde
"Gimme about 10 seconds to think for a minute..."

Re: Gnus does not handle quoting in quoted strings correctly.

[Katsumi Yamaoka]

>>>>> In <E1GIr5K-0000LA-L1@fencepost.gnu.org> Richard Stallman wrote:

> Please DTRT.

Done.  The changes I made in the Gnus CVS will be merged into
Emacs soon.

>>>>> In <m2ejuyt0vv.fsf@kenny.sha-bang.de> Sascha Wilde wrote:

> if you have a mail with

> From: "foo \[bar\]" <foo@bar.invalid>

> in the headers, gnus displays the from literately as

> foo \[bar\]

> but according to RfC822/2822 "\" is used in quoted strings as special
> quoting character and is always (no matter if necessary or not)
> semantically "invisible".  So it should read:

> foo [bar]

Gnus does so now in both article and summary buffers.  Please
note that `\"' and `\\' in a quoted string are unchanged since
`"' and `\' are not qtext (cf. RFC2822 section 3.2.5).

> The other way around, when writing display names in quoted strings, no
> quoting is done even when it would be needed.  Writing a message with

> From: "foo/bar" <foo@bar.invalid>

> the From: header is copied verbatim to the outgoing mail, but should
> get quoted like:

> From: "foo//bar" <foo@bar.invalid>

Well, I made changes in Gnus' encoder so that it might convert
"foo/bar" into "foo\/bar".  Is it correct?

;; I will have no time to work on Gnus until next week.  Sorry.

Re: Gnus does not handle quoting in quoted strings correctly.

[Richard Stallman]

    Done.  The changes I made in the Gnus CVS will be merged into
    Emacs soon.

We want to start pretesting ASAP, but maybe we should wait for this.
Can you say when you can do this?  How long would we need to wait?

Re: Gnus does not handle quoting in quoted strings correctly.

[Katsumi Yamaoka]

>>>>> In <E1GJGuc-0002fL-RQ@fencepost.gnu.org>
>>>>>	Richard Stallman <rms@gnu.org> wrote:

>     Done.  The changes I made in the Gnus CVS will be merged into
>     Emacs soon.

> We want to start pretesting ASAP, but maybe we should wait for this.
> Can you say when you can do this?  How long would we need to wait?

Sync'ing Gnus CVS and Emacs CVS is done periodically by Miles
Bader AFAIK.  Miles, could you please do it now specially?

Re: Gnus does not handle quoting in quoted strings correctly.

[Miles Bader]

Katsumi Yamaoka <yamaoka@jpl.org> writes:
> Sync'ing Gnus CVS and Emacs CVS is done periodically by Miles
> Bader AFAIK.  Miles, could you please do it now specially?

Ok, it's done.

-miles

-- 
o The existentialist, not having a pillow, goes everywhere with the book by
  Sullivan, _I am going to spit on your graves_.

Re: Gnus does not handle quoting in quoted strings correctly.

[Sascha Wilde]

Katsumi Yamaoka <yamaoka@jpl.org> wrote:

>>>>>> In <m2ejuyt0vv.fsf@kenny.sha-bang.de> Sascha Wilde wrote:
>
>> if you have a mail with
>
>> From: "foo \[bar\]" <foo@bar.invalid>
>
>> in the headers, gnus displays the from literately as
>
>> foo \[bar\]
>
>> but according to RfC822/2822 "\" is used in quoted strings as special
>> quoting character and is always (no matter if necessary or not)
>> semantically "invisible".  So it should read:
>
>> foo [bar]
>
> Gnus does so now in both article and summary buffers.  Please
> note that `\"' and `\\' in a quoted string are unchanged since
> `"' and `\' are not qtext (cf. RFC2822 section 3.2.5).

Sorry, I'm not sure if I understand you right.  

Right now gnus (from CVS emacs) does not display quoted pairs in
quoted strings as expected.

Pleas note that according to RfC2822 3.2.5 quoted strings are defined
as:

quoted-string   =       [CFWS]
                        DQUOTE *([FWS] qcontent) [FWS] DQUOTE
                        [CFWS]

where qcontent is

qcontent        =       qtext / quoted-pair

so quoted pairs have to be handled in quoted strings and `\"' and
`\\' are legal quoted pairs (you can quote any "text", see RfC2822
3.2.1 and 3.2.2) so they must be displayed as `"' and `\'.

>> The other way around, when writing display names in quoted strings, no
>> quoting is done even when it would be needed.  Writing a message with
>
>> From: "foo/bar" <foo@bar.invalid>
>
>> the From: header is copied verbatim to the outgoing mail, but should
>> get quoted like:
>
>> From: "foo//bar" <foo@bar.invalid>
>
> Well, I made changes in Gnus' encoder so that it might convert
> "foo/bar" into "foo\/bar".  Is it correct?

oops, sorry, that was a typo by me...

the examples should read 

 From: "foo\bar" <foo@bar.invalid>

and

 From: "foo\\bar" <foo@bar.invalid>

of cause.  What I wanted to say is: Any non qtext character in quoted
strings in the input should be quoted by gnus encoder.

cheers
sascha
-- 
Sascha Wilde
"C++ : an octopus made by nailing extra legs onto a dog"

Re: Gnus does not handle quoting in quoted strings correctly.

[Richard Stallman]

Thanks.

Re: Gnus does not handle quoting in quoted strings correctly.

[Katsumi Yamaoka]

>>>>> In <m2mz9is6vt.fsf@kenny.sha-bang.de>
>>>>>	Sascha Wilde <wilde@sha-bang.de> wrote:

>> note that `\"' and `\\' in a quoted string are unchanged since
>> `"' and `\' are not qtext (cf. RFC2822 section 3.2.5).

> Sorry, I'm not sure if I understand you right.

I'm not quite sure of it but FLIM's decoder (which SEMI MUAs
such as Wanderlust uses) does so, and I followed it.

> Right now gnus (from CVS emacs) does not display quoted pairs in
> quoted strings as expected.

> Pleas note that according to RfC2822 3.2.5 quoted strings are defined
> as:

> quoted-string   =       [CFWS]
>                         DQUOTE *([FWS] qcontent) [FWS] DQUOTE
>                         [CFWS]

> where qcontent is

> qcontent        =       qtext / quoted-pair

> so quoted pairs have to be handled in quoted strings and `\"' and
> `\\' are legal quoted pairs (you can quote any "text", see RfC2822
> 3.2.1 and 3.2.2) so they must be displayed as `"' and `\'.

I agree that `\\' in a quoted string is decoded into `\'.
However, I don't think it is useful that

"foo \"bar\""  is decoded into  "foo "bar""  or  foo "bar"

because it may appear in the From header and may be used in the
To header when replying.

WDYT?

Re: Gnus does not handle quoting in quoted strings correctly.

[Sascha Wilde]

Katsumi Yamaoka <yamaoka@jpl.org> wrote:
>>>>>> In <m2mz9is6vt.fsf@kenny.sha-bang.de>
>>>>>>	Sascha Wilde <wilde@sha-bang.de> wrote:
[...]
>> so quoted pairs have to be handled in quoted strings and `\"' and
>> `\\' are legal quoted pairs (you can quote any "text", see RfC2822
>> 3.2.1 and 3.2.2) so they must be displayed as `"' and `\'.
>
> I agree that `\\' in a quoted string is decoded into `\'.
> However, I don't think it is useful that
>
> "foo \"bar\""  is decoded into  "foo "bar""  or  foo "bar"
>
> because it may appear in the From header and may be used in the
> To header when replying.

I do agree that decoding `\"' would be a problem in such situations,
as it wouldn't be possible determine unambiguously how to re-encode
the string.

I'd think the ideal solution would be to only decode `\"' for display
(in summary and article buffers) but to keep it quoted when used to
generate a new message (like a reply for instance).

I don't know if this can be done easily or if the current code base
doesn't allow to make this difference -- if it's to hard to do I would
vote for leaving `\"' quoted...

cheers
sascha
-- 
Sascha Wilde : xedit? Das sieht zwar wie vi aus als könne es nix, aber im
             : Gegensatz zu vi kann es wirklich nix und nix ist noch geschönt!
             : (Michael Core in dafc)

Re: Gnus does not handle quoting in quoted strings correctly.

[Katsumi Yamaoka]

>>>>> In <m2r6yshpbu.fsf@kenny.sha-bang.de> Sascha Wilde wrote:

> Katsumi Yamaoka <yamaoka@jpl.org> wrote:

>> I agree that `\\' in a quoted string is decoded into `\'.

I've made a change further in both the v5-10 branch and the
trunk so that the decoder might do so.  Although that behavior
is different from FLIM which I always relied on as a model, it
doesn't seem to cause any harm as far as it is done in Gnus
because `\' in a quoted string is now encoded into `\\' when
sending a message.

Miles, I'm sorry to bother you again but please merge it to the
Emacs CVS.  Thanks in advance.

>> However, I don't think it is useful that
>>
>> "foo \"bar\""  is decoded into  "foo "bar""  or  foo "bar"
>>
>> because it may appear in the From header and may be used in the
>> To header when replying.

> I do agree that decoding `\"' would be a problem in such situations,
> as it wouldn't be possible determine unambiguously how to re-encode
> the string.

> I'd think the ideal solution would be to only decode `\"' for display
> (in summary and article buffers) but to keep it quoted when used to
> generate a new message (like a reply for instance).

I agree.  And it is supposed possible to achieve.  However, I
think we should not start such a development in the Emacs CVS
*now* since appearing of `\'s doesn't look so serious to me and
it seems to require that it is tested thoroughly.

> I don't know if this can be done easily or if the current code base
> doesn't allow to make this difference -- if it's to hard to do I would
> vote for leaving `\"' quoted...

I don't yet know how it is easy or difficult.  So, I vote for it
as well.