smtpmail support of unicode passwords

smtpmail support of unicode passwords

[Garreau, Alexandre]

[-- Attachment #1.1: Type: text/plain, Size: 288 bytes --]

Hello, smtpmail fails to send mails when the SMTP user password contain
utf8 chars, since it seems base64-encode-string doesn’t support
multibyte encoding, it prevented me to start using Gnus for some months
before I understood what was happening.

Here’s attached the patch.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2.1: Patch for smtpmail --]
[-- Type: text/x-diff, Size: 545 bytes --]

diff -Naur emacs-old/lisp/mail/smtpmail.el emacs/lisp/mail/smtpmail.el
--- emacs-old/lisp/mail/smtpmail.el	2014-04-13 18:08:54.210814676 +0000
+++ emacs/lisp/mail/smtpmail.el	2014-04-13 18:06:07.502802860 +0000
@@ -580,7 +580,7 @@
       (smtpmail-command-or-throw
        process
        (concat "AUTH PLAIN "
-	       (base64-encode-string (concat "\0" user "\0" password) t))
+	       (base64-encode-string (string-as-unibyte (concat "\0" user "\0" password)) t))
        235))
      (t
       (error "Mechanism %s not implemented" mech)))))

[-- Attachment #1.2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 948 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 948 bytes --]

Re: smtpmail support of unicode passwords

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Date: Sun, 13 Apr 2014 20:27:49 +0200
> 
> Hello, smtpmail fails to send mails when the SMTP user password contain
> utf8 chars, since it seems base64-encode-string doesn’t support
> multibyte encoding, it prevented me to start using Gnus for some months
> before I understood what was happening.
> 
> Here’s attached the patch.

I don't think string-as-unibyte is correct here, please try

  (encode-coding-string (concat "\0" user "\0" password) 'utf-8)

instead.

Re: smtpmail support of unicode passwords

[Stefan Monnier]

> I don't think string-as-unibyte is correct here,

Indeed, it's pretty much never correct.

> please try
>   (encode-coding-string (concat "\0" user "\0" password) 'utf-8)
> instead.

I think locale-coding-system would make more sense.


        Stefan

Re: smtpmail support of unicode passwords

[Stephen J. Turnbull]

Stefan Monnier writes:
 > > I don't think string-as-unibyte is correct here,
 > 
 > Indeed, it's pretty much never correct.
 > 
 > > please try
 > >   (encode-coding-string (concat "\0" user "\0" password) 'utf-8)
 > > instead.
 > 
 > I think locale-coding-system would make more sense.

This password is for a remote system, no?  You need special support
for non-ASCII here.  smtpmail-password-coding-system, I think.

Re: smtpmail support of unicode passwords

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: "Garreau\, Alexandre" <galex-713@galex-713.eu>, emacs-devel@gnu.org
> Date: Sun, 13 Apr 2014 21:18:51 -0400
> 
> > please try
> >   (encode-coding-string (concat "\0" user "\0" password) 'utf-8)
> > instead.
> 
> I think locale-coding-system would make more sense.

My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
expert in this area.

Re: smtpmail support of unicode passwords

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,
>     "Garreau, Alexandre" <galex-713@galex-713.eu>,
>     emacs-devel@gnu.org
> Date: Mon, 14 Apr 2014 14:50:19 +0900
> 
> Stefan Monnier writes:
>  > > please try
>  > >   (encode-coding-string (concat "\0" user "\0" password) 'utf-8)
>  > > instead.
>  > 
>  > I think locale-coding-system would make more sense.
> 
> This password is for a remote system, no?  You need special support
> for non-ASCII here.  smtpmail-password-coding-system, I think.

There's no such variable in Emacs's smtpmail.el.

Re: smtpmail support of unicode passwords

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
 > expert in this area.

By relevant RFCs I assume you mean RFC 4616 (aka SASL "PLAIN")?

Indeed, the receiving server is going to expect that the string is
valid UTF-8.  That doesn't necessarily mean that Emacs has the string
as characters in internal encoding, appropriate to be encoded via the
utf-8 coding system before being sent on the wire.  It might also have
it as bytes read from a file using the binary codec (the main use case
I'm worried about), or the user may have learned to type it that way,
etc.

Of course if you know you have a string decoded using binary, you can
translate it using utf-8 before sending it back through utf-8 and then
onto the wire.

I guess it's actually quite likely that the binary file contains
Latin-N or some form of Cyrillic, etc, so if needed the decode-encode
cycle is probably the way to go.  So you may as well hard-code utf-8
here after all.

Re: smtpmail support of unicode passwords

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,
>     galex-713@galex-713.eu,
>     emacs-devel@gnu.org
> Date: Mon, 14 Apr 2014 21:14:50 +0900
> 
> Eli Zaretskii writes:
> 
>  > My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
>  > expert in this area.
> 
> By relevant RFCs I assume you mean RFC 4616 (aka SASL "PLAIN")?

Yes.

> Indeed, the receiving server is going to expect that the string is
> valid UTF-8.  That doesn't necessarily mean that Emacs has the string
> as characters in internal encoding, appropriate to be encoded via the
> utf-8 coding system before being sent on the wire.  It might also have
> it as bytes read from a file using the binary codec (the main use case
> I'm worried about), or the user may have learned to type it that way,
> etc.

If a unibyte string gets to that place, it's a separate bug that needs
to be fixed separately.  As a matter of principle, any human-readable
text read into Emacs should be decoded, rather than left as raw bytes.

When the user types anything, Emacs decodes input before storing it in
buffers or strings.  And if the password is produced by a function,
that function should make sure it produces a decoded multibyte string.

So yes, I think we want to depend on having multibyte strings there.

Re: smtpmail support of unicode passwords

[Stefan Monnier]

> My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
> expert in this area.

Oh, so it's actually standardized?  Wonderful, utf-8 it is then, much
better, indeed!


        Stefan

Re: smtpmail support of unicode passwords

[Harald Hanche-Olsen]

[Stefan Monnier <monnier@IRO.UMontreal.CA> (2014-04-14 12:55:17 UTC)]

> > My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
> > expert in this area.
> 
> Oh, so it's actually standardized?  Wonderful, utf-8 it is then, much
> better, indeed!

It is with some trepidation I notice that RFC 4616 does not mention
normalization at all. Are "élan" and "élan" the same password or not?
Should the latter (NFD) be normalized to the former (NFC) or not? Or
the other way around?

(In case it is not obvious, é is U+00E9 LATIN SMALL LETTER E WITH
ACUTE, while the NFD form contains an ordinary U+0045 LATIN CAPITAL
LETTER E followed by U+0301 COMBINING ACUTE ACCENT)

I am unaware of any standardization in this area, but I bring it up so
you can at least be aware of the issue, even if it's not clear how to
handle it.

– Harald

PS. Relevant reference: http://unicode.org/reports/tr15/

Re: smtpmail support of unicode passwords

[Eli Zaretskii]

> Date: Mon, 14 Apr 2014 16:57:02 +0200 (CEST)
> From: Harald Hanche-Olsen <hanche@math.ntnu.no>
> 
> It is with some trepidation I notice that RFC 4616 does not mention
> normalization at all. Are "élan" and "élan" the same password or not?
> Should the latter (NFD) be normalized to the former (NFC) or not? Or
> the other way around?
> 
> (In case it is not obvious, é is U+00E9 LATIN SMALL LETTER E WITH
> ACUTE, while the NFD form contains an ordinary U+0045 LATIN CAPITAL
> LETTER E followed by U+0301 COMBINING ACUTE ACCENT)
> 
> I am unaware of any standardization in this area, but I bring it up so
> you can at least be aware of the issue, even if it's not clear how to
> handle it.

If someone can find out what is TRT here, Emacs can normalize if
needed.

Re: smtpmail support of unicode passwords

[Harald Hanche-Olsen]

[Eli Zaretskii <eliz@gnu.org> (2014-04-14 15:36:23 UTC)]

> > Date: Mon, 14 Apr 2014 16:57:02 +0200 (CEST)
> > From: Harald Hanche-Olsen <hanche@math.ntnu.no>
> > 
> > It is with some trepidation I notice that RFC 4616 does not mention
> > normalization at all. Are "élan" and "élan" the same password or not?
> > Should the latter (NFD) be normalized to the former (NFC) or not? Or
> > the other way around?
> 
> If someone can find out what is TRT here, Emacs can normalize if
> needed.

Poking around a little more in RFC4616, I see a reference to SASLprep
(RFC 4013) and StringPrep (RFC 3454). Apparently, the correct
normalization is NFKC; however, that is considered optional, and
reading between the lines a bit, I think this normalization, if done
at allo, is intended to happen in the server – just prior to computing
the hash.

If I have that right, TRT on the client side is to just pass the
string through unchanged. Which is a relief.

Apologies for not doing this small bit of research before raising the
issue.

– Harald

Re: smtpmail support of unicode passwords

[Stephen J. Turnbull]

Harald Hanche-Olsen writes:

 > Poking around a little more in RFC4616, I see a reference to SASLprep
 > (RFC 4013) and StringPrep (RFC 3454). Apparently, the correct
 > normalization is NFKC; however, that is considered optional,

No, it's not.  RFC 4013, sec. 2.2, uses SHALL, RFC 4616, sec. 2, uses
RECOMMENDED, and RFC 4954, sec. 4 uses SHOULD (with a prospective
upgrade to MUST).  So technically, you can use something else, but
you'd better have a REALLY GOOD (in the RFC 2119 sense ;-) reason.

But it's not Emacs's problem:

 > reading between the lines a bit, I think this normalization, if done
 > at allo, is intended to happen in the server – just prior to computing
 > the hash.

As usual the RFCs leave rather wide discretion to the implementers.
But in the case of SMTP I think it would be insane for servers to fail
to normalize.

Re: smtpmail support of unicode passwords

[Eli Zaretskii]

> Date: Mon, 14 Apr 2014 18:55:29 +0200 (CEST)
> Cc: emacs-devel@gnu.org
> From: Harald Hanche-Olsen <hanche@math.ntnu.no>
> 
> Poking around a little more in RFC4616, I see a reference to SASLprep
> (RFC 4013) and StringPrep (RFC 3454). Apparently, the correct
> normalization is NFKC; however, that is considered optional, and
> reading between the lines a bit, I think this normalization, if done
> at allo, is intended to happen in the server – just prior to computing
> the hash.
> 
> If I have that right, TRT on the client side is to just pass the
> string through unchanged. Which is a relief.

Thanks for looking into it.

> Apologies for not doing this small bit of research before raising the
> issue.

No need for apologies.