all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* smtpmail support of unicode passwords
@ 2014-04-13 18:27 Garreau, Alexandre
  2014-04-13 19:11 ` Eli Zaretskii
  0 siblings, 1 reply; 14+ messages in thread
From: Garreau, Alexandre @ 2014-04-13 18:27 UTC (permalink / raw)
  To: emacs-devel


[-- Attachment #1.1: Type: text/plain, Size: 288 bytes --]

Hello, smtpmail fails to send mails when the SMTP user password contain
utf8 chars, since it seems base64-encode-string doesn’t support
multibyte encoding, it prevented me to start using Gnus for some months
before I understood what was happening.

Here’s attached the patch.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2.1: Patch for smtpmail --]
[-- Type: text/x-diff, Size: 545 bytes --]

diff -Naur emacs-old/lisp/mail/smtpmail.el emacs/lisp/mail/smtpmail.el
--- emacs-old/lisp/mail/smtpmail.el	2014-04-13 18:08:54.210814676 +0000
+++ emacs/lisp/mail/smtpmail.el	2014-04-13 18:06:07.502802860 +0000
@@ -580,7 +580,7 @@
       (smtpmail-command-or-throw
        process
        (concat "AUTH PLAIN "
-	       (base64-encode-string (concat "\0" user "\0" password) t))
+	       (base64-encode-string (string-as-unibyte (concat "\0" user "\0" password)) t))
        235))
      (t
       (error "Mechanism %s not implemented" mech)))))

[-- Attachment #1.2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 948 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 948 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-13 18:27 smtpmail support of unicode passwords Garreau, Alexandre
@ 2014-04-13 19:11 ` Eli Zaretskii
  2014-04-14  1:18   ` Stefan Monnier
  0 siblings, 1 reply; 14+ messages in thread
From: Eli Zaretskii @ 2014-04-13 19:11 UTC (permalink / raw)
  To: Garreau, Alexandre; +Cc: emacs-devel

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Date: Sun, 13 Apr 2014 20:27:49 +0200
> 
> Hello, smtpmail fails to send mails when the SMTP user password contain
> utf8 chars, since it seems base64-encode-string doesn’t support
> multibyte encoding, it prevented me to start using Gnus for some months
> before I understood what was happening.
> 
> Here’s attached the patch.

I don't think string-as-unibyte is correct here, please try

  (encode-coding-string (concat "\0" user "\0" password) 'utf-8)

instead.




^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-13 19:11 ` Eli Zaretskii
@ 2014-04-14  1:18   ` Stefan Monnier
  2014-04-14  5:50     ` Stephen J. Turnbull
  2014-04-14  7:44     ` Eli Zaretskii
  0 siblings, 2 replies; 14+ messages in thread
From: Stefan Monnier @ 2014-04-14  1:18 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: Garreau, Alexandre, emacs-devel

> I don't think string-as-unibyte is correct here,

Indeed, it's pretty much never correct.

> please try
>   (encode-coding-string (concat "\0" user "\0" password) 'utf-8)
> instead.

I think locale-coding-system would make more sense.


        Stefan



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14  1:18   ` Stefan Monnier
@ 2014-04-14  5:50     ` Stephen J. Turnbull
  2014-04-14  8:18       ` Eli Zaretskii
  2014-04-14  7:44     ` Eli Zaretskii
  1 sibling, 1 reply; 14+ messages in thread
From: Stephen J. Turnbull @ 2014-04-14  5:50 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Eli Zaretskii, Garreau, Alexandre, emacs-devel

Stefan Monnier writes:
 > > I don't think string-as-unibyte is correct here,
 > 
 > Indeed, it's pretty much never correct.
 > 
 > > please try
 > >   (encode-coding-string (concat "\0" user "\0" password) 'utf-8)
 > > instead.
 > 
 > I think locale-coding-system would make more sense.

This password is for a remote system, no?  You need special support
for non-ASCII here.  smtpmail-password-coding-system, I think.




^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14  1:18   ` Stefan Monnier
  2014-04-14  5:50     ` Stephen J. Turnbull
@ 2014-04-14  7:44     ` Eli Zaretskii
  2014-04-14 12:14       ` Stephen J. Turnbull
  2014-04-14 12:55       ` Stefan Monnier
  1 sibling, 2 replies; 14+ messages in thread
From: Eli Zaretskii @ 2014-04-14  7:44 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: galex-713, emacs-devel

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: "Garreau\, Alexandre" <galex-713@galex-713.eu>, emacs-devel@gnu.org
> Date: Sun, 13 Apr 2014 21:18:51 -0400
> 
> > please try
> >   (encode-coding-string (concat "\0" user "\0" password) 'utf-8)
> > instead.
> 
> I think locale-coding-system would make more sense.

My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
expert in this area.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14  5:50     ` Stephen J. Turnbull
@ 2014-04-14  8:18       ` Eli Zaretskii
  0 siblings, 0 replies; 14+ messages in thread
From: Eli Zaretskii @ 2014-04-14  8:18 UTC (permalink / raw)
  To: Stephen J. Turnbull; +Cc: monnier, galex-713, emacs-devel

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,
>     "Garreau, Alexandre" <galex-713@galex-713.eu>,
>     emacs-devel@gnu.org
> Date: Mon, 14 Apr 2014 14:50:19 +0900
> 
> Stefan Monnier writes:
>  > > please try
>  > >   (encode-coding-string (concat "\0" user "\0" password) 'utf-8)
>  > > instead.
>  > 
>  > I think locale-coding-system would make more sense.
> 
> This password is for a remote system, no?  You need special support
> for non-ASCII here.  smtpmail-password-coding-system, I think.

There's no such variable in Emacs's smtpmail.el.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14  7:44     ` Eli Zaretskii
@ 2014-04-14 12:14       ` Stephen J. Turnbull
  2014-04-14 12:27         ` Eli Zaretskii
  2014-04-14 12:55       ` Stefan Monnier
  1 sibling, 1 reply; 14+ messages in thread
From: Stephen J. Turnbull @ 2014-04-14 12:14 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: Stefan Monnier, galex-713, emacs-devel

Eli Zaretskii writes:

 > My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
 > expert in this area.

By relevant RFCs I assume you mean RFC 4616 (aka SASL "PLAIN")?

Indeed, the receiving server is going to expect that the string is
valid UTF-8.  That doesn't necessarily mean that Emacs has the string
as characters in internal encoding, appropriate to be encoded via the
utf-8 coding system before being sent on the wire.  It might also have
it as bytes read from a file using the binary codec (the main use case
I'm worried about), or the user may have learned to type it that way,
etc.

Of course if you know you have a string decoded using binary, you can
translate it using utf-8 before sending it back through utf-8 and then
onto the wire.

I guess it's actually quite likely that the binary file contains
Latin-N or some form of Cyrillic, etc, so if needed the decode-encode
cycle is probably the way to go.  So you may as well hard-code utf-8
here after all.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14 12:14       ` Stephen J. Turnbull
@ 2014-04-14 12:27         ` Eli Zaretskii
  0 siblings, 0 replies; 14+ messages in thread
From: Eli Zaretskii @ 2014-04-14 12:27 UTC (permalink / raw)
  To: Stephen J. Turnbull; +Cc: monnier, galex-713, emacs-devel

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,
>     galex-713@galex-713.eu,
>     emacs-devel@gnu.org
> Date: Mon, 14 Apr 2014 21:14:50 +0900
> 
> Eli Zaretskii writes:
> 
>  > My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
>  > expert in this area.
> 
> By relevant RFCs I assume you mean RFC 4616 (aka SASL "PLAIN")?

Yes.

> Indeed, the receiving server is going to expect that the string is
> valid UTF-8.  That doesn't necessarily mean that Emacs has the string
> as characters in internal encoding, appropriate to be encoded via the
> utf-8 coding system before being sent on the wire.  It might also have
> it as bytes read from a file using the binary codec (the main use case
> I'm worried about), or the user may have learned to type it that way,
> etc.

If a unibyte string gets to that place, it's a separate bug that needs
to be fixed separately.  As a matter of principle, any human-readable
text read into Emacs should be decoded, rather than left as raw bytes.

When the user types anything, Emacs decodes input before storing it in
buffers or strings.  And if the password is produced by a function,
that function should make sure it produces a decoded multibyte string.

So yes, I think we want to depend on having multibyte strings there.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14  7:44     ` Eli Zaretskii
  2014-04-14 12:14       ` Stephen J. Turnbull
@ 2014-04-14 12:55       ` Stefan Monnier
  2014-04-14 14:57         ` Harald Hanche-Olsen
  1 sibling, 1 reply; 14+ messages in thread
From: Stefan Monnier @ 2014-04-14 12:55 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: galex-713, emacs-devel

> My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
> expert in this area.

Oh, so it's actually standardized?  Wonderful, utf-8 it is then, much
better, indeed!


        Stefan



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14 12:55       ` Stefan Monnier
@ 2014-04-14 14:57         ` Harald Hanche-Olsen
  2014-04-14 15:36           ` Eli Zaretskii
  0 siblings, 1 reply; 14+ messages in thread
From: Harald Hanche-Olsen @ 2014-04-14 14:57 UTC (permalink / raw)
  To: emacs-devel

[Stefan Monnier <monnier@IRO.UMontreal.CA> (2014-04-14 12:55:17 UTC)]

> > My reading if the relevant RFCs is that UTF-8 must be used, but I'm no
> > expert in this area.
> 
> Oh, so it's actually standardized?  Wonderful, utf-8 it is then, much
> better, indeed!

It is with some trepidation I notice that RFC 4616 does not mention
normalization at all. Are "élan" and "élan" the same password or not?
Should the latter (NFD) be normalized to the former (NFC) or not? Or
the other way around?

(In case it is not obvious, é is U+00E9 LATIN SMALL LETTER E WITH
ACUTE, while the NFD form contains an ordinary U+0045 LATIN CAPITAL
LETTER E followed by U+0301 COMBINING ACUTE ACCENT)

I am unaware of any standardization in this area, but I bring it up so
you can at least be aware of the issue, even if it's not clear how to
handle it.

– Harald

PS. Relevant reference: http://unicode.org/reports/tr15/



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14 14:57         ` Harald Hanche-Olsen
@ 2014-04-14 15:36           ` Eli Zaretskii
  2014-04-14 16:55             ` Harald Hanche-Olsen
  0 siblings, 1 reply; 14+ messages in thread
From: Eli Zaretskii @ 2014-04-14 15:36 UTC (permalink / raw)
  To: Harald Hanche-Olsen; +Cc: emacs-devel

> Date: Mon, 14 Apr 2014 16:57:02 +0200 (CEST)
> From: Harald Hanche-Olsen <hanche@math.ntnu.no>
> 
> It is with some trepidation I notice that RFC 4616 does not mention
> normalization at all. Are "élan" and "élan" the same password or not?
> Should the latter (NFD) be normalized to the former (NFC) or not? Or
> the other way around?
> 
> (In case it is not obvious, é is U+00E9 LATIN SMALL LETTER E WITH
> ACUTE, while the NFD form contains an ordinary U+0045 LATIN CAPITAL
> LETTER E followed by U+0301 COMBINING ACUTE ACCENT)
> 
> I am unaware of any standardization in this area, but I bring it up so
> you can at least be aware of the issue, even if it's not clear how to
> handle it.

If someone can find out what is TRT here, Emacs can normalize if
needed.




^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14 15:36           ` Eli Zaretskii
@ 2014-04-14 16:55             ` Harald Hanche-Olsen
  2014-04-14 18:03               ` Stephen J. Turnbull
  2014-04-14 21:01               ` Eli Zaretskii
  0 siblings, 2 replies; 14+ messages in thread
From: Harald Hanche-Olsen @ 2014-04-14 16:55 UTC (permalink / raw)
  To: eliz; +Cc: emacs-devel

[Eli Zaretskii <eliz@gnu.org> (2014-04-14 15:36:23 UTC)]

> > Date: Mon, 14 Apr 2014 16:57:02 +0200 (CEST)
> > From: Harald Hanche-Olsen <hanche@math.ntnu.no>
> > 
> > It is with some trepidation I notice that RFC 4616 does not mention
> > normalization at all. Are "élan" and "élan" the same password or not?
> > Should the latter (NFD) be normalized to the former (NFC) or not? Or
> > the other way around?
> 
> If someone can find out what is TRT here, Emacs can normalize if
> needed.

Poking around a little more in RFC4616, I see a reference to SASLprep
(RFC 4013) and StringPrep (RFC 3454). Apparently, the correct
normalization is NFKC; however, that is considered optional, and
reading between the lines a bit, I think this normalization, if done
at allo, is intended to happen in the server – just prior to computing
the hash.

If I have that right, TRT on the client side is to just pass the
string through unchanged. Which is a relief.

Apologies for not doing this small bit of research before raising the
issue.

– Harald



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14 16:55             ` Harald Hanche-Olsen
@ 2014-04-14 18:03               ` Stephen J. Turnbull
  2014-04-14 21:01               ` Eli Zaretskii
  1 sibling, 0 replies; 14+ messages in thread
From: Stephen J. Turnbull @ 2014-04-14 18:03 UTC (permalink / raw)
  To: Harald Hanche-Olsen; +Cc: eliz, emacs-devel

Harald Hanche-Olsen writes:

 > Poking around a little more in RFC4616, I see a reference to SASLprep
 > (RFC 4013) and StringPrep (RFC 3454). Apparently, the correct
 > normalization is NFKC; however, that is considered optional,

No, it's not.  RFC 4013, sec. 2.2, uses SHALL, RFC 4616, sec. 2, uses
RECOMMENDED, and RFC 4954, sec. 4 uses SHOULD (with a prospective
upgrade to MUST).  So technically, you can use something else, but
you'd better have a REALLY GOOD (in the RFC 2119 sense ;-) reason.

But it's not Emacs's problem:

 > reading between the lines a bit, I think this normalization, if done
 > at allo, is intended to happen in the server – just prior to computing
 > the hash.

As usual the RFCs leave rather wide discretion to the implementers.
But in the case of SMTP I think it would be insane for servers to fail
to normalize.






^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: smtpmail support of unicode passwords
  2014-04-14 16:55             ` Harald Hanche-Olsen
  2014-04-14 18:03               ` Stephen J. Turnbull
@ 2014-04-14 21:01               ` Eli Zaretskii
  1 sibling, 0 replies; 14+ messages in thread
From: Eli Zaretskii @ 2014-04-14 21:01 UTC (permalink / raw)
  To: Harald Hanche-Olsen; +Cc: emacs-devel

> Date: Mon, 14 Apr 2014 18:55:29 +0200 (CEST)
> Cc: emacs-devel@gnu.org
> From: Harald Hanche-Olsen <hanche@math.ntnu.no>
> 
> Poking around a little more in RFC4616, I see a reference to SASLprep
> (RFC 4013) and StringPrep (RFC 3454). Apparently, the correct
> normalization is NFKC; however, that is considered optional, and
> reading between the lines a bit, I think this normalization, if done
> at allo, is intended to happen in the server – just prior to computing
> the hash.
> 
> If I have that right, TRT on the client side is to just pass the
> string through unchanged. Which is a relief.

Thanks for looking into it.

> Apologies for not doing this small bit of research before raising the
> issue.

No need for apologies.




^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2014-04-14 21:01 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-04-13 18:27 smtpmail support of unicode passwords Garreau, Alexandre
2014-04-13 19:11 ` Eli Zaretskii
2014-04-14  1:18   ` Stefan Monnier
2014-04-14  5:50     ` Stephen J. Turnbull
2014-04-14  8:18       ` Eli Zaretskii
2014-04-14  7:44     ` Eli Zaretskii
2014-04-14 12:14       ` Stephen J. Turnbull
2014-04-14 12:27         ` Eli Zaretskii
2014-04-14 12:55       ` Stefan Monnier
2014-04-14 14:57         ` Harald Hanche-Olsen
2014-04-14 15:36           ` Eli Zaretskii
2014-04-14 16:55             ` Harald Hanche-Olsen
2014-04-14 18:03               ` Stephen J. Turnbull
2014-04-14 21:01               ` Eli Zaretskii

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.