bug#14380: [gmane.emacs.bugs] bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

bug#14380: [gmane.emacs.bugs] bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

On Sat, May 18, 2013 at 1:33 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> Sorry, I forgot to CC the two of you and the post just went to the bug
> list.  I didn't know if you'd see it so this is just a courtesy CC.
>
> Ted
>
>
>
> ---------- Forwarded message ----------
> From: Ted Zlatanov <tzz@lifelogs.com>
> To: 14380@debbugs.gnu.org
> Cc:
> Date: Fri, 17 May 2013 09:12:05 -0400
> Subject: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32
> On Fri, 10 May 2013 21:44:12 +0100 João Távora <joaotavora@gmail.com> wrote:
>
> JT> Honestly I thought builtin tls was some kind of tls library in elisp.
> JT> Needing external libraries doesn't make it very builtin :-), but
> JT> that's just my opinion.
>
> Implementing TLS in ELisp is possible, sure, but it would be extremely
> slow and there's considerable risk in reimplementing that protocol.  We
> chose to use GnuTLS instead, which is a fast reliable C implementation
> and is available for W32 and all our other platforms.

lol, I wasn't suggesting implementing it in elisp at all, I just read
built-in and assumed in was either statically linked in C or elisp. The
point it that needing external libraries which are not always bundled
doesn't make it very "builtin". But naming things, along with cache
invalidation,
is a a hard problem in programming :-)

> I've seen dozens of bugs related to "almost working" external TLS
> binaries on all platforms.

Yes, but have you looked closely at this particular one? The point is rather
to increase robustness. That is, `open-tls-stream` could/should promise
to cleanup the process buffer of its handshake garbage, so that future
functions that use that resource don't see it and don't get confused by it.

I'm assuming they don't need to see it, I might be wrong.

But if I'm right and that fix is performed then you've effectively extended
"imap just works" the set of W32 emacs users who type "M-x gnus" on a
vanilla emacs in a system with some cygwin installation in PATH. Maybe it's
a small set but I'm in it (when I'm at work).

> GnuTLS integration with Emacs.  My vote is to require GnuTLS with Emacs
> and to only support it, but there are some questions there, mainly for
> W32 and Mac OS X: do we auto-update GnuTLS?  What happens when the
> GnuTLS we install conflicts with another system install?  And so on...

That's all fine, I guess. I vote for that too :-)

Thanks,
João

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

(CC to emacs-devel as I think this discussion is relevant there)

On Sat, 18 May 2013 14:05:47 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> The point [is] that needing external libraries which are not always
JT> bundled doesn't make it very "builtin".

I'm not bringing GnuTLS into the Emacs source tree, which is the only
other way to make it built-in functionality.  I understand there are
issues with external dependencies and in fact I asked that we bundle the
GnuTLS W32 DLLs with the W32 Emacs builds.  That led to a long
discussions about how that makes security our responsibility and how we
then need to deal with GnuTLS updates, and I didn't have a strong desire
to become a W32 distribution expert since I barely know that platform.
No one else picked it up, and there we are with "install it yourself" as
the recommended way to get GnuTLS to work on W32.

>> I've seen dozens of bugs related to "almost working" external TLS
>> binaries on all platforms.

JT> Yes, but have you looked closely at this particular one? The point is rather
JT> to increase robustness. That is, `open-tls-stream` could/should promise
JT> to cleanup the process buffer of its handshake garbage, so that future
JT> functions that use that resource don't see it and don't get confused by it.

JT> I'm assuming they don't need to see it, I might be wrong.

I'm not able to fix this bug or work on bugs in the external SSL/TLS support.

JT> But if I'm right and that fix is performed then you've effectively extended
JT> "imap just works" the set of W32 emacs users who type "M-x gnus" on a
JT> vanilla emacs in a system with some cygwin installation in PATH. Maybe it's
JT> a small set but I'm in it (when I'm at work).

Wouldn't you rather get GnuTLS to work by default?  Otherwise we serve
the use case "I have no secure transport, so let me use a hack by
default."

>> GnuTLS integration with Emacs.  My vote is to require GnuTLS with Emacs
>> and to only support it, but there are some questions there, mainly for
>> W32 and Mac OS X: do we auto-update GnuTLS?  What happens when the
>> GnuTLS we install conflicts with another system install?  And so on...

JT> That's all fine, I guess. I vote for that too :-)

The big problem for me is that I don't have the time or platform
knowledge to write a GnuTLS auto-installer and updater for those two
problematic platforms.  The GnuTLS developers don't want to provide this
service either.  Who will be responsible to it?  What happens when a
security vulnerability hits the DLLs we distribute with Emacs?

My proposal would be to push out the next Emacs bundled with the latest
GnuTLS DLLs, only support GnuTLS, provide users with instructions on
updating them, and treat GnuTLS vulnerabilities as Emacs
vulnerabilities.  This is not ideal but IMO better than the current
situation.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

(CC to emacs-devel as I think this discussion is relevant there)

On Sat, 18 May 2013 14:05:47 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> The point [is] that needing external libraries which are not always
JT> bundled doesn't make it very "builtin".

I'm not bringing GnuTLS into the Emacs source tree, which is the only
other way to make it built-in functionality.  I understand there are
issues with external dependencies and in fact I asked that we bundle the
GnuTLS W32 DLLs with the W32 Emacs builds.  That led to a long
discussions about how that makes security our responsibility and how we
then need to deal with GnuTLS updates, and I didn't have a strong desire
to become a W32 distribution expert since I barely know that platform.
No one else picked it up, and there we are with "install it yourself" as
the recommended way to get GnuTLS to work on W32.

>> I've seen dozens of bugs related to "almost working" external TLS
>> binaries on all platforms.

JT> Yes, but have you looked closely at this particular one? The point is rather
JT> to increase robustness. That is, `open-tls-stream` could/should promise
JT> to cleanup the process buffer of its handshake garbage, so that future
JT> functions that use that resource don't see it and don't get confused by it.

JT> I'm assuming they don't need to see it, I might be wrong.

I'm not able to fix this bug or work on bugs in the external SSL/TLS support.

JT> But if I'm right and that fix is performed then you've effectively extended
JT> "imap just works" the set of W32 emacs users who type "M-x gnus" on a
JT> vanilla emacs in a system with some cygwin installation in PATH. Maybe it's
JT> a small set but I'm in it (when I'm at work).

Wouldn't you rather get GnuTLS to work by default?  Otherwise we serve
the use case "I have no secure transport, so let me use a hack by
default."

>> GnuTLS integration with Emacs.  My vote is to require GnuTLS with Emacs
>> and to only support it, but there are some questions there, mainly for
>> W32 and Mac OS X: do we auto-update GnuTLS?  What happens when the
>> GnuTLS we install conflicts with another system install?  And so on...

JT> That's all fine, I guess. I vote for that too :-)

The big problem for me is that I don't have the time or platform
knowledge to write a GnuTLS auto-installer and updater for those two
problematic platforms.  The GnuTLS developers don't want to provide this
service either.  Who will be responsible to it?  What happens when a
security vulnerability hits the DLLs we distribute with Emacs?

My proposal would be to push out the next Emacs bundled with the latest
GnuTLS DLLs, only support GnuTLS, provide users with instructions on
updating them, and treat GnuTLS vulnerabilities as Emacs
vulnerabilities.  This is not ideal but IMO better than the current
situation.

Ted

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

On Sun, May 19, 2013 at 4:17 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> GnuTLS W32 DLLs with the W32 Emacs builds.  That led to a long
> discussions about how that makes security our responsibility and how we

I see. Indeed, bundling security stuff with your app is increasing
its responsibility manifold.

> Wouldn't you rather get GnuTLS to work by default?  Otherwise we serve
> the use case "I have no secure transport, so let me use a hack by
> default."

I don't understand. What is the hack here? External binary for TLS?
But yes, GnuTLS by default is certainly better...

> service either.  Who will be responsible to it?  What happens when a
> security vulnerability hits the DLLs we distribute with Emacs?
>
> My proposal would be to push out the next Emacs bundled with the latest
> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
> updating them, and treat GnuTLS vulnerabilities as Emacs
> vulnerabilities.  This is not ideal but IMO better than the current
> situation.

... but then you have all these headaches.

The fix I proposed aims for the status quo, that is: make external
TLS binary support slightly more robust. My test case is even smaller:

* W32
* cygwin carrying the responsibility burden
* vanilla emacs working with tls/imap/gnus.

Thanks for the time spent in analysing this,
-- 
João Távora

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

On Sun, May 19, 2013 at 4:17 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> GnuTLS W32 DLLs with the W32 Emacs builds.  That led to a long
> discussions about how that makes security our responsibility and how we

I see. Indeed, bundling security stuff with your app is increasing
its responsibility manifold.

> Wouldn't you rather get GnuTLS to work by default?  Otherwise we serve
> the use case "I have no secure transport, so let me use a hack by
> default."

I don't understand. What is the hack here? External binary for TLS?
But yes, GnuTLS by default is certainly better...

> service either.  Who will be responsible to it?  What happens when a
> security vulnerability hits the DLLs we distribute with Emacs?
>
> My proposal would be to push out the next Emacs bundled with the latest
> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
> updating them, and treat GnuTLS vulnerabilities as Emacs
> vulnerabilities.  This is not ideal but IMO better than the current
> situation.

... but then you have all these headaches.

The fix I proposed aims for the status quo, that is: make external
TLS binary support slightly more robust. My test case is even smaller:

* W32
* cygwin carrying the responsibility burden
* vanilla emacs working with tls/imap/gnus.

Thanks for the time spent in analysing this,
-- 
João Távora

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 19 May 2013 12:45:12 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, 14380@debbugs.gnu.org, emacs-devel@gnu.org
> 
> The fix I proposed aims for the status quo, that is: make external
> TLS binary support slightly more robust.

I already said at lest twice in this thread: THIS WON'T WORK on
Windows (except in Cygwin Emacs).  The communications between the
external TLS client and Emacs are via signals, which aren't really
supported by Windows.  Solving this was one of the main reasons for
incorporating GnuTLS into Emacs.

I don't really understand what are we still discussing here.  Let me
describe how the current situation looks from my POV:

  . Emacs can be built with GnuTLS support if GnuTLS is installed on
    the end-user's machine, and that end user builds her own Emacs.
    This is the same as on Unix.  I hope no one will say this is "not
    built-in".

  . Windows users get special treatment in that precompiled binaries
    of Emacs are available for those who cannot or won't build their
    own.  These precompiled binaries are built with GnuTLS support to
    begin with.

  . As yet another bonus for Windows users, Emacs will happily start
    and run even if GnuTLS is not found on the end-user's machine;
    however, TLS will not be available in that case, of course (Emacs
    will announce that if required to use TLS).

So now you tell me how come these two bonuses are somehow regarded as
deficiencies?  Would it be better not to produce binaries at all, or
let them abort with a fatal error if GnuTLS is not installed?

Installing GnuTLS boils down to unzipping a single zip archive.  How
hard can that be for someone who uses Emacs??

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

On Sun, May 19, 2013 at 4:44 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: João Távora <joaotavora@gmail.com>
>> Date: Sun, 19 May 2013 12:45:12 +0100
>> Cc: Eli Zaretskii <eliz@gnu.org>, 14380@debbugs.gnu.org, emacs-devel@gnu.org
>>
>> The fix I proposed aims for the status quo, that is: make external
>> TLS binary support slightly more robust.
>
> I already said at lest twice in this thread: THIS WON'T WORK on
> Windows (except in Cygwin Emacs).  The communications between the

Look, there's no need to shout. I'm not using Cygwin emacs, I'm using
regular W32
binaries and am not even sure what tls binary emacs found or how. It
appears to be:

"openssl s_client -connect imaps.mycompany.com:993 -no_ssl2 -ign_eof"

My analysis of the code of `network-stream-open-tls' revealed (as do
the comments)
that it tries to cleanup the process buffer of previous garbage left there by
`open-tls-stream` (who nonetheless tries to place point correctly in
the process
buffer)

I'm **guessing** "openssl" is a cygwin binary, I didn't even check that.
I **reported** a bug since I considered unexpected behaviour occurred even with
the cleanest of "emacs -Q" run.
I **suggested** a fix because of two reasons: (1) I tried it and it
worked and has
been working since (2) in the context of the interaction between the
two functions
`network-stream-open-tls' and `open-tls-stream' it seemed reasonable
that the latter
cleans up after itself.

Maybe, in my reduced usage of gnus, I haven't gotten to a situation
where things would
break because of signal handling or whatever. Lucky me.

When things do break, I'll happily unzip dlls, I have nothing against that.

Thanks for all the info, feel free to close the bug if you haven't already
Over and out,
João

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 19 May 2013 18:57:43 +0100
> Cc: tzz@lifelogs.com, 14380@debbugs.gnu.org
> 
> On Sun, May 19, 2013 at 4:44 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: João Távora <joaotavora@gmail.com>
> >> Date: Sun, 19 May 2013 12:45:12 +0100
> >> Cc: Eli Zaretskii <eliz@gnu.org>, 14380@debbugs.gnu.org, emacs-devel@gnu.org
> >>
> >> The fix I proposed aims for the status quo, that is: make external
> >> TLS binary support slightly more robust.
> >
> > I already said at lest twice in this thread: THIS WON'T WORK on
> > Windows (except in Cygwin Emacs).  The communications between the
> 
> Look, there's no need to shout.

I'm sorry, but after saying that twice, what other methods do I have
to make sure I'm heard?

> I'm not using Cygwin emacs, I'm using regular W32 binaries

That's exactly what I was talking about: an external tls client will
not work with a native w32 Emacs.

> When things do break, I'll happily unzip dlls, I have nothing against that.

I suggest you do this without waiting for things to break.  The
built-in GnuTLS support does work on Windows, there are several happy
users of it here.  Why have a partial solution when you can have a
complete one?

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sun, 19 May 2013 12:45:12 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> On Sun, May 19, 2013 at 4:17 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Wouldn't you rather get GnuTLS to work by default?  Otherwise we serve
>> the use case "I have no secure transport, so let me use a hack by
>> default."

JT> I don't understand. What is the hack here? External binary for TLS?

Using an external binary to transport SSL or TLS is a hack IMO.

>> My proposal would be to push out the next Emacs bundled with the latest
>> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
>> updating them, and treat GnuTLS vulnerabilities as Emacs
>> vulnerabilities.  This is not ideal but IMO better than the current
>> situation.

JT> ... but then you have all these headaches.

It's a headache I'm willing to endure for the sake of Emacs users.

The alternative, which João is enduring now, is to punt the problem.

This is a question for the Emacs maintainers: do you agree with me on
the above plan?  It would mean changing the way Mac OS X and W32 Emacs
builds are distributed, to include the GnuTLS libraries with the build,
and we'd have to implement a way (perhaps through the ELPA) to
distribute updates to these libraries.

JT> The fix I proposed aims for the status quo, that is: make external
JT> TLS binary support slightly more robust. My test case is even smaller:

JT> * W32
JT> * cygwin carrying the responsibility burden
JT> * vanilla emacs working with tls/imap/gnus.

Did you propose a patch?  I would commit a patch but can't write it
despite your great description of the problem.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sun, 19 May 2013 12:45:12 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> On Sun, May 19, 2013 at 4:17 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Wouldn't you rather get GnuTLS to work by default?  Otherwise we serve
>> the use case "I have no secure transport, so let me use a hack by
>> default."

JT> I don't understand. What is the hack here? External binary for TLS?

Using an external binary to transport SSL or TLS is a hack IMO.

>> My proposal would be to push out the next Emacs bundled with the latest
>> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
>> updating them, and treat GnuTLS vulnerabilities as Emacs
>> vulnerabilities.  This is not ideal but IMO better than the current
>> situation.

JT> ... but then you have all these headaches.

It's a headache I'm willing to endure for the sake of Emacs users.

The alternative, which João is enduring now, is to punt the problem.

This is a question for the Emacs maintainers: do you agree with me on
the above plan?  It would mean changing the way Mac OS X and W32 Emacs
builds are distributed, to include the GnuTLS libraries with the build,
and we'd have to implement a way (perhaps through the ELPA) to
distribute updates to these libraries.

JT> The fix I proposed aims for the status quo, that is: make external
JT> TLS binary support slightly more robust. My test case is even smaller:

JT> * W32
JT> * cygwin carrying the responsibility burden
JT> * vanilla emacs working with tls/imap/gnus.

Did you propose a patch?  I would commit a patch but can't write it
despite your great description of the problem.

Ted

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Juanma Barranquero]

On Mon, May 20, 2013 at 1:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:

> It would mean changing the way Mac OS X and W32 Emacs
> builds are distributed, to include the GnuTLS libraries with the build,
> and we'd have to implement a way (perhaps through the ELPA) to
> distribute updates to these libraries.

Why would that be a good idea now if it wasn't seen as such before?

    J

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Juanma Barranquero]

On Mon, May 20, 2013 at 1:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:

> It would mean changing the way Mac OS X and W32 Emacs
> builds are distributed, to include the GnuTLS libraries with the build,
> and we'd have to implement a way (perhaps through the ELPA) to
> distribute updates to these libraries.

Why would that be a good idea now if it wasn't seen as such before?

    J

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 04:08:57 +0200 Juanma Barranquero <lekktu@gmail.com> wrote: 

JB> On Mon, May 20, 2013 at 1:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> It would mean changing the way Mac OS X and W32 Emacs
>> builds are distributed, to include the GnuTLS libraries with the build,
>> and we'd have to implement a way (perhaps through the ELPA) to
>> distribute updates to these libraries.

JB> Why would that be a good idea now if it wasn't seen as such before?

Because we've had years of experience supporting GnuTLS, we're moving to
GnuTLS 3.x soon (there were no objections to my proposal for that), the
ELPA has matured and requires a secure transport[1], and we've heard
many times from our users on those two platforms that we should.

The current situation is worse: users are on their own and the GnuTLS
developers have not given them a way to auto-update or indicated
interest in doing so (I asked last year IIRC, and see
http://josefsson.org/gnutls4win/)

Ted

[1] I also proposed the ELPA as the way to distribute GnuTLS updates.
There's a separate thread about signing ELPA packages which would let us
avoid using a broken GnuTLS to pull its own updates, but that's not
essential to my proposal.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Juanma Barranquero]

On Mon, May 20, 2013 at 11:58 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:

> Because we've had years of experience supporting GnuTLS, we're moving to
> GnuTLS 3.x soon (there were no objections to my proposal for that), the
> ELPA has matured and requires a secure transport[1], and we've heard
> many times from our users on those two platforms that we should.

It still seems to me like a pretty serious change in our distribution
model, just because of GnuTLS on Windows and Mac OS.

I agree with Eli that installing GnuTLS from a zip file is easy enough.

   J

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 17:21:47 +0200 Juanma Barranquero <lekktu@gmail.com> wrote: 

JB> On Mon, May 20, 2013 at 11:58 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Because we've had years of experience supporting GnuTLS, we're moving to
>> GnuTLS 3.x soon (there were no objections to my proposal for that), the
>> ELPA has matured and requires a secure transport[1], and we've heard
>> many times from our users on those two platforms that we should.

JB> It still seems to me like a pretty serious change in our distribution
JB> model, just because of GnuTLS on Windows and Mac OS.

I think in the distribution of libraries on those two platforms we are
not even in the same millennium as most other software there.  I'm
willing to live with the non-security libraries like libxml (although
those can be compromised too).  But we should at least make an effort to
tell the user they are out of date, even if we do nothing more.

JB> I agree with Eli that installing GnuTLS from a zip file is easy enough.

OK, 2 votes against and 1 in favor.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 24 May 2013 15:32:40 -0400
> 
> I think in the distribution of libraries on those two platforms we are
> not even in the same millennium as most other software there.

You (or someone else) are welcome to take my ports and package them as
you see fit.  If you do a good job, I will even point to those
packages in nt/INSTALL etc.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Fri, 24 May 2013 22:50:01 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Fri, 24 May 2013 15:32:40 -0400
>> 
>> I think in the distribution of libraries on those two platforms we are
>> not even in the same millennium as most other software there.

EZ> You (or someone else) are welcome to take my ports and package them as
EZ> you see fit.  If you do a good job, I will even point to those
EZ> packages in nt/INSTALL etc.

Your work is very good and OK with me (although I can work with you to
automate the builds).  I am talking about distributing these binary
libraries better, especially critical updates.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 24 May 2013 17:55:16 -0400
> 
> EZ> You (or someone else) are welcome to take my ports and package them as
> EZ> you see fit.  If you do a good job, I will even point to those
> EZ> packages in nt/INSTALL etc.
> 
> Your work is very good and OK with me (although I can work with you to
> automate the builds).  I am talking about distributing these binary
> libraries better, especially critical updates.

That's exactly what I was talking about, too: take the port packaged
as drop-in zip files, and package it in some other way you consider to
be better.

As for critical updates, if you point me to the place or places where
these are described, I might consider whether it's time to update the
port.  (Please don't just point me to the gnutls-devel list, because I
read that, and have yet to see something like what you seem to allude
to.)

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sat, 25 May 2013 09:42:56 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Fri, 24 May 2013 17:55:16 -0400
>> 
EZ> You (or someone else) are welcome to take my ports and package them as
EZ> you see fit.  If you do a good job, I will even point to those
EZ> packages in nt/INSTALL etc.
>> 
>> Your work is very good and OK with me (although I can work with you to
>> automate the builds).  I am talking about distributing these binary
>> libraries better, especially critical updates.

EZ> That's exactly what I was talking about, too: take the port packaged
EZ> as drop-in zip files, and package it in some other way you consider to
EZ> be better.

That's great!  I'd love to make it a GNU ELPA package to be installed or
updated like any other such package.

EZ> As for critical updates, if you point me to the place or places where
EZ> these are described, I might consider whether it's time to update the
EZ> port.  (Please don't just point me to the gnutls-devel list, because I
EZ> read that, and have yet to see something like what you seem to allude
EZ> to.)

I would assume you want to automate the builds to some degree, but I
don't mind packaging whatever you produce.  Have you documented the
process anywhere so I can follow it if you can't do it?

There's several lists where critical vulnerabilities are disclosed but I
don't follow them all; currently this seems like the best way:
http://web.nvd.nist.gov/view/vuln/search-results?query=gnutls&search_type=all&cves=on&uscert_ta=on&uscert_vn=on&oval_query=on

Following gnutls-devel is probably sufficient for now and we can tell
the GnuTLS developers we are interested in early disclosure.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 25 May 2013 18:18:06 -0400
> 
> I would assume you want to automate the builds to some degree

When it comes to Windows, I don't believe in automated builds.

> Have you documented the process anywhere so I can follow it if you
> can't do it?

It's just the usual configure and make; the "interesting" parts begin
when one or both fail.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sun, 26 May 2013 05:52:52 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 25 May 2013 18:18:06 -0400
>> 
>> I would assume you want to automate the builds to some degree

EZ> When it comes to Windows, I don't believe in automated builds.

Thank you.  I won't pursue an automated W32 build further, just a
distribution mechanism for GnuTLS and possibly other libraries.

>> Have you documented the process anywhere so I can follow it if you
>> can't do it?

EZ> It's just the usual configure and make; the "interesting" parts begin
EZ> when one or both fail.

OK, got it.

Thanks
Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 25 May 2013 18:18:06 -0400
> 
> There's several lists where critical vulnerabilities are disclosed but I
> don't follow them all; currently this seems like the best way:
> http://web.nvd.nist.gov/view/vuln/search-results?query=gnutls&search_type=all&cves=on&uscert_ta=on&uscert_vn=on&oval_query=on

This seems to be very old.  This page looks much more relevant:

   http://gnutls.org/security.html

According to the latter, we are in good shape with 3.0.9.  I'll
probably look into updating to 3.2.x, once it stabilizes.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sun, 26 May 2013 22:59:05 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 25 May 2013 18:18:06 -0400
>> 
>> There's several lists where critical vulnerabilities are disclosed but I
>> don't follow them all; currently this seems like the best way:
>> http://web.nvd.nist.gov/view/vuln/search-results?query=gnutls&search_type=all&cves=on&uscert_ta=on&uscert_vn=on&oval_query=on

EZ> This seems to be very old.  This page looks much more relevant:

EZ>    http://gnutls.org/security.html

There's also BUGTRAQ at http://www.securityfocus.com/archive/1

I don't know of a single place to track vulnerabilities, honestly.  They
seem to be all over the place.

EZ> According to the latter, we are in good shape with 3.0.9.  I'll
EZ> probably look into updating to 3.2.x, once it stabilizes.

OK.  That's good to know, and thank you for looking into it.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Stefan Monnier]

> those can be compromised too).  But we should at least make an effort to
> tell the user they are out of date, even if we do nothing more.
[...]
JB> I agree with Eli that installing GnuTLS from a zip file is easy enough.
> OK, 2 votes against and 1 in favor.

I haven't read this thread, so if you want my opinion, please explain
concisely what it's about (e.g. in which circumstance should we tell
the user (s)he's out of date).


        Stefan

GnuTLS updates proposal (was: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32)

[Ted Zlatanov]

On Fri, 24 May 2013 15:51:08 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> those can be compromised too).  But we should at least make an effort to
>> tell the user they are out of date, even if we do nothing more.
SM> [...]
JB> I agree with Eli that installing GnuTLS from a zip file is easy enough.
>> OK, 2 votes against and 1 in favor.

SM> I haven't read this thread, so if you want my opinion, please explain
SM> concisely what it's about (e.g. in which circumstance should we tell
SM> the user (s)he's out of date).

Generally library updates are announced on specialized lists that users
don't follow and a developer picks it up and pushes it out to the users.

I am proposing that GnuTLS updates should be pushed out this way (which
would also cover installation, so the user doesn't have to get it from a
ZIP file on W32 or from God knows where on Mac OS X...  Homebrew or Fink
or something else) instead of putting that responsibility on the user.

We could use the GNU ELPA as the transport (once it has package
signing).  I can be responsible for following the GnuTLS announcements.

This is not typical for Emacs so I've brought it up for discussion
instead of just doing the work.

So far Eli and Juanma are against my proposal.

Ted

Re: GnuTLS updates proposal

[Stefan Monnier]

> I am proposing that GnuTLS updates should be pushed out this way (which
> would also cover installation, so the user doesn't have to get it from a
> ZIP file on W32 or from God knows where on Mac OS X...  Homebrew or Fink
> or something else) instead of putting that responsibility on the user.

That sounds OK so far.

> We could use the GNU ELPA as the transport (once it has package
> signing).  I can be responsible for following the GnuTLS announcements.

Now I lost you.  How did we go from "gnutls" to "we"?
Shouldn't some gnutls guys take care of that?


        Stefan

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Fri, 24 May 2013 21:52:14 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I am proposing that GnuTLS updates should be pushed out this way (which
>> would also cover installation, so the user doesn't have to get it from a
>> ZIP file on W32 or from God knows where on Mac OS X...  Homebrew or Fink
>> or something else) instead of putting that responsibility on the user.

SM> That sounds OK so far.

Wonderful.

>> We could use the GNU ELPA as the transport (once it has package
>> signing).  I can be responsible for following the GnuTLS announcements.

SM> Now I lost you.  How did we go from "gnutls" to "we"?
SM> Shouldn't some gnutls guys take care of that?

So far they have not and it's not a priority to them.  Eli did a lot of
work cleaning up the code and getting it to compile--it's thanks to him
that Emacs has a recent GnuTLS DLL.

If you want, I can ask again and see what they say now... I'm not
optimistic.

So, as far as building the GnuTLS DLLs, I'm hopeful we'll find a way to
automate that but not hopeful that it will happen on the GnuTLS
maintainers' side.  Building them on Mac OS X is trivial.  This is the
"hard work" part and Eli already pointed out we'll need volunteers and
resources.

Whoever provides the actual DLLs, there has to be a package maintainer
for the GNU ELPA package of GnuTLS; I will do it unless there's someone
else excited by the prospect of tracking security announcements.  Plus,
of course, we can't rely on the GNU ELPA for library updates until the
package signing work is done.

Ted

Re: GnuTLS updates proposal

[Stefan Monnier]

SM> Now I lost you.  How did we go from "gnutls" to "we"?
SM> Shouldn't some gnutls guys take care of that?

> So far they have not and it's not a priority to them.  Eli did a lot of
> work cleaning up the code and getting it to compile--it's thanks to him
> that Emacs has a recent GnuTLS DLL.

So, why should this fall on us?  I mean, is Emacs the only program
using gnutls?
If so, why?  what do other programs use?
If not, what do the other programs do about it?

> So, as far as building the GnuTLS DLLs, I'm hopeful we'll find a way to
> automate that but not hopeful that it will happen on the GnuTLS
> maintainers' side.  Building them on Mac OS X is trivial.  This is the
> "hard work" part and Eli already pointed out we'll need volunteers and
> resources.

Ah, so the problem is not only for Windows but also for Mac OS X?

> Whoever provides the actual DLLs, there has to be a package maintainer
> for the GNU ELPA package of GnuTLS;

As a user, I wouldn't like it for a program such as Emacs to decide it
will take over responsibility for a generic library used by other
programs, so I find the idea of distributing it via GNU ELPA rather odd.


        Stefan

Re: GnuTLS updates proposal

[Stephen J. Turnbull]

Stefan Monnier writes:

 > SM> Now I lost you.  How did we go from "gnutls" to "we"?
 > SM> Shouldn't some gnutls guys take care of that?

No, why should they?  Emacs doesn't provide Debian or Red Hat
packages, the downstream does that.  When Emacs is downstream, it
will be expected to do the work, if anybody does.

 > > So far they have not and it's not a priority to them.  Eli did a lot of
 > > work cleaning up the code and getting it to compile--it's thanks to him
 > > that Emacs has a recent GnuTLS DLL.
 > 
 > So, why should this fall on us?  I mean, is Emacs the only program
 > using gnutls?

Wrong question.  The question is does Emacs (which is a collective
work drawing on many packages not developed here) use GnuTLS?  If so,
is there an existing distribution of GnuTLS that "enough" users use so
that it's preferable to ask the users who don't have it to download it
themselves, rather than include it in the Emacs (or ELPA) distribution?

The debate here is about the value of "enough".  Ted wants much closer
to 100% than Eli and Juanma think necessary.

 > If so, why?  what do other programs use?

Most programs I know of use OpenSSL.  There are political problems,
and maybe license problems (the EAY clause) with Emacs doing that (but
see the list of OpenSSL users below).  From my MacPorts installation:

MacPorts 14:40$ port dependents gnutls
gnutls has no dependents.

(My build of the Emacs trunk depends on gnutls, but not the MacPorts
version I currently have installed.)

MacPorts 15:03$ port dependents openssl
cmake depends on openssl
curl depends on openssl
cyrus-sasl2 depends on openssl
git-core depends on openssl
gnome-vfs depends on openssl
kerberos5 depends on openssl
neon depends on openssl
openldap depends on openssl
openssh depends on openssl
python26 depends on openssl
python27 depends on openssl
python32 depends on openssl
python33 depends on openssl
qt4-mac depends on openssl
serf0 depends on openssl
serf1 depends on openssl

Note that these are only the installed packages (so it's biased to my
taste), and there are several cases of multiple versions (but the
ratio is nevertheless damning).  Note that even gnome-vfs depends on
OpenSSL.

 > If not, what do the other programs do about it?

Avoid it, apparently.

 > Ah, so the problem is not only for Windows but also for Mac OS X?

The problem exists for all OSes.  Eg, Debian stable is probably more
out of date than either of the above (until sometime in the next few
days if I'm reading the tea leaves right).

However, aggregating Mac and Windows here should be done with great
care (see below).

 > As a user, I wouldn't like it for a program such as Emacs to decide it
 > will take over responsibility for a generic library used by other
 > programs, so I find the idea of distributing it via GNU ELPA rather odd.

That's because you don't use Windows.  Windows users are used to that,
and in fact prefer it (if done right), because many programs have
problems with the "wrong" version of certain libraries, so they bundle
them.

Mac, however, is much closer to the *nix ideal of one instance per
package, so I suspect a lot of Mac users would prefer to use their
system version of gnutls, whereever it comes from (mine comes from
MacPorts).  But "DLL Hell" is a pain on many *nix systems as well (eg,
my Gentoo update is currently blocked because SBCL demands one version
of certain libraries, while Gentoo wants to install more recent ones).
It is also common in Python applications (MacPorts installs a separate
compilation of each library I install for all interpreter versions I
have -- I need 3 to support various applications, plus the most recent
release for development).

Heck, Emacs (until the advent of ELPA) was perhaps the premier example
of this behavior: distribute all the Lisp anyone could ever want, so
that the developers could have a free hand in refactoring.  XEmacs
proposed (on several occasions) that one of the existing systems
(XEmacs and the OSU repository are two I remember, the latter not
really being a package system but rather a classification system) but
were refused.  The reason given was that all Lisp was part of Emacs,
and Emacs could only afford to support one version of the language at
a time so a package system made no sense: it would constrain the
language development too much.

That said, indeed there are Mac users who expect full-blown
installers.  But I would expect them, by the same token, to mostly be
using Aquamacs.  In that case you can leave it up to the Aquamacs
maintainer(s), and the issue is restricted to Windows.

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Sat, 25 May 2013 00:47:41 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> Now I lost you.  How did we go from "gnutls" to "we"?
SM> Shouldn't some gnutls guys take care of that?

>> So far they have not and it's not a priority to them.  Eli did a lot of
>> work cleaning up the code and getting it to compile--it's thanks to him
>> that Emacs has a recent GnuTLS DLL.

SM> So, why should this fall on us?  I mean, is Emacs the only program
SM> using gnutls?
SM> If so, why?  what do other programs use?
SM> If not, what do the other programs do about it?

GnuTLS is not a popular solution.  OpenSSL is far, far more popular and
well supported on W32 and other platforms.  So there isn't a good sample
size to answer that question.

In addition, Emacs itself is not an application, it's more of a platform
IMO.  That makes it a little more responsible for keeping its libraries
up to date than most applications, again IMHO.

>> So, as far as building the GnuTLS DLLs, I'm hopeful we'll find a way to
>> automate that but not hopeful that it will happen on the GnuTLS
>> maintainers' side.  Building them on Mac OS X is trivial.  This is the
>> "hard work" part and Eli already pointed out we'll need volunteers and
>> resources.

SM> Ah, so the problem is not only for Windows but also for Mac OS X?

Yes, because on Mac OS X there is no authoritative package manager
either.

>> Whoever provides the actual DLLs, there has to be a package maintainer
>> for the GNU ELPA package of GnuTLS;

SM> As a user, I wouldn't like it for a program such as Emacs to decide it
SM> will take over responsibility for a generic library used by other
SM> programs, so I find the idea of distributing it via GNU ELPA rather odd.

If it was a program, sure.  The closest analogue to Emacs on W32 is
Cygwin as a whole, which does provide its own updates.

I realize my proposal is not the way we've always done things and seems
odd and unusual.  My goal right now is to collect votes for or against
my proposal, especially from the maintainers.  If everyone but me says
it's a bad idea, then it probably is.

Ted

Re: GnuTLS updates proposal

[Richard Stallman]

Emacs cannot link with OpenSSL due to license incompatibility.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Sun, 26 May 2013 00:51:01 -0400 Richard Stallman <rms@gnu.org> wrote: 

RS> Emacs cannot link with OpenSSL due to license incompatibility.

Of course.  Stefan was asking how other applications deal with the same
problem, and I mentioned OpenSSL because it's the closest analogue to
GnuTLS.

Ted

Re: GnuTLS updates proposal (was: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32)

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 24 May 2013 17:53:36 -0400
> 
> So far Eli and Juanma are against my proposal.

Correction: they think that the current arrangement, whereby users are
pointed to specific distributions of GnuTLS and other 3rd party
packages, are adequate enough, and any additional effort is something
the Emacs developers are unlikely to be able to sustain.

Re: GnuTLS updates proposal

[Stefan Monnier]

> Correction: they think that the current arrangement, whereby users are
> pointed to specific distributions of GnuTLS and other 3rd party
> packages, are adequate enough, and any additional effort is something
> the Emacs developers are unlikely to be able to sustain.

I sure won't spend any time on this, indeed, and I hope you don't
either.  But I guess if someone wants to provide some Elisp code that
eases up the installation (taking care of downloading the right DLL
(which I guess might depend on the Emacs version and even the specific
build) and installing it at the right place), that would be OK and could
very well fit in GNU ELPA (I don't think I'd want the DLL itself to be
there).  Hopefully such code would handle not just gnutls but also the
other libraries, like libxml.


        Stefan

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Sat, 25 May 2013 03:10:36 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Correction: they think that the current arrangement, whereby users are
>> pointed to specific distributions of GnuTLS and other 3rd party
>> packages, are adequate enough, and any additional effort is something
>> the Emacs developers are unlikely to be able to sustain.

SM> I sure won't spend any time on this, indeed, and I hope you don't
SM> either.  But I guess if someone wants to provide some Elisp code that
SM> eases up the installation (taking care of downloading the right DLL
SM> (which I guess might depend on the Emacs version and even the specific
SM> build) and installing it at the right place), that would be OK and could
SM> very well fit in GNU ELPA (I don't think I'd want the DLL itself to be
SM> there).  Hopefully such code would handle not just gnutls but also the
SM> other libraries, like libxml.

Sounds good, I'll put it on my TODO list after ELPA package signing.
Thank you for looking at my proposal.

Ted

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Sun, 26 May 2013 17:34:10 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Sat, 25 May 2013 03:10:36 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>>> Correction: they think that the current arrangement, whereby users are
>>> pointed to specific distributions of GnuTLS and other 3rd party
>>> packages, are adequate enough, and any additional effort is something
>>> the Emacs developers are unlikely to be able to sustain.

SM> I sure won't spend any time on this, indeed, and I hope you don't
SM> either.  But I guess if someone wants to provide some Elisp code that
SM> eases up the installation (taking care of downloading the right DLL
SM> (which I guess might depend on the Emacs version and even the specific
SM> build) and installing it at the right place), that would be OK and could
SM> very well fit in GNU ELPA (I don't think I'd want the DLL itself to be
SM> there).  Hopefully such code would handle not just gnutls but also the
SM> other libraries, like libxml.

TZ> Sounds good, I'll put it on my TODO list after ELPA package signing.
TZ> Thank you for looking at my proposal.

Update: I talked to the GnuTLS developers.  They now have W32 binaries
but do *not* guarantee anything about them except their internal tests
have passed, and don't consider W32 an important platform.  So the Emacs
W32 integration could fail and they will still release those DLLs.  But
at least the W32 build is considered somewhat important by them.

It seems that it's better to keep relying on Eli and myself as the
build+QA team for the Emacs side, and to keep using Eli's binaries, with
me as the fallback if he can't produce them.  So basically this does not
change the plans we discussed earlier, but we now have more assurance of
a working W32 baseline from the GnuTLS developers.

Ted

Re: GnuTLS updates proposal (was: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32)

[Juanma Barranquero]

On Sat, May 25, 2013 at 8:39 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> Correction: they think that the current arrangement, whereby users are
> pointed to specific distributions of GnuTLS and other 3rd party
> packages, are adequate enough, and any additional effort is something
> the Emacs developers are unlikely to be able to sustain.

That's exactly my opinion, yeah.

   J

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Mon, 20 May 2013 05:58:40 -0400
> 
> On Mon, 20 May 2013 04:08:57 +0200 Juanma Barranquero <lekktu@gmail.com> wrote: 
> 
> JB> On Mon, May 20, 2013 at 1:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> >> It would mean changing the way Mac OS X and W32 Emacs
> >> builds are distributed, to include the GnuTLS libraries with the build,
> >> and we'd have to implement a way (perhaps through the ELPA) to
> >> distribute updates to these libraries.
> 
> JB> Why would that be a good idea now if it wasn't seen as such before?
> 
> Because we've had years of experience supporting GnuTLS, we're moving to
> GnuTLS 3.x soon

What do you mean by "moving to GnuTLS 3.x soon"?  For Windows users,
the recommended GnuTLS port is of v3.0.9; isn't that "3.x" already?

> ELPA has matured and requires a secure transport[1], and we've heard
> many times from our users on those two platforms that we should.
> 
> The current situation is worse: users are on their own and the GnuTLS
> developers have not given them a way to auto-update or indicated
> interest in doing so (I asked last year IIRC, and see
> http://josefsson.org/gnutls4win/)
> 
> Ted
> 
> [1] I also proposed the ELPA as the way to distribute GnuTLS updates.
> There's a separate thread about signing ELPA packages which would let us
> avoid using a broken GnuTLS to pull its own updates, but that's not
> essential to my proposal.

Sorry, I'm confused: are you talking about securing Emacs or about
securing ELPA?  You mix these two (which are quite different issues
with very different implications) in ways that make it hard to
understand what are you saying.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 19:17:45 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Mon, 20 May 2013 05:58:40 -0400
>> 
>> On Mon, 20 May 2013 04:08:57 +0200 Juanma Barranquero <lekktu@gmail.com> wrote: 
>> 
JB> On Mon, May 20, 2013 at 1:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> >> It would mean changing the way Mac OS X and W32 Emacs
>> >> builds are distributed, to include the GnuTLS libraries with the build,
>> >> and we'd have to implement a way (perhaps through the ELPA) to
>> >> distribute updates to these libraries.
>> 
JB> Why would that be a good idea now if it wasn't seen as such before?
>> 
>> Because we've had years of experience supporting GnuTLS, we're moving to
>> GnuTLS 3.x soon

EZ> What do you mean by "moving to GnuTLS 3.x soon"?  For Windows users,
EZ> the recommended GnuTLS port is of v3.0.9; isn't that "3.x" already?

We use the 2.x API which works with 3.x.

>> [1] I also proposed the ELPA as the way to distribute GnuTLS updates.
>> There's a separate thread about signing ELPA packages which would let us
>> avoid using a broken GnuTLS to pull its own updates, but that's not
>> essential to my proposal.

EZ> Sorry, I'm confused: are you talking about securing Emacs or about
EZ> securing ELPA?  You mix these two (which are quite different issues
EZ> with very different implications) in ways that make it hard to
EZ> understand what are you saying.

There is nothing about securing Emacs itself here (that topic includes
secure credential storage and such, which are not relevant here).

I'm talking about two things:

1) distributing GnuTLS with Emacs and also distributing updates to it.

2) possibly using the ELPA to distribute the GnuTLS updates as a package
that's signed.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

On Mon, May 20, 2013 at 12:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>
> Using an external binary to transport SSL or TLS is a hack IMO.

I see, and reading a bit of tls.el it makes sense. It does all this
crazy regexp searching... This is a hack in my book too, albeit one
that's been working fine,

> Did you propose a patch?  I would commit a patch but can't write it
> despite your great description of the problem.
>

Here is a patch that should be equivalent to the defadvice I'm using.
As I said, it works for me. Also I didn't have a VC copy of emacs so I
used  `diff-buffer-with-file'

diff -u -L /usr/local/share/emacs/24.3/lisp/net/tls.el.gz -L
\#\<buffer\ tls.el.gz\> /tmp/jka-com1909LVh
/tmp/buffer-content-1909lpt
--- /usr/local/share/emacs/24.3/lisp/net/tls.el.gz
+++ #<buffer tls.el.gz>
@@ -286,7 +286,11 @@
 			     (format "Host name in certificate doesn't \
 match `%s'. Connect anyway? " host))))))
 	(setq done nil)
-	(delete-process process)))
+	(delete-process process))
+      ;; delete all the informational messages that could
+      ;; confuse futures users of `buffer'
+      ;;
+      (delete-region (point-min) (point)))
     (message "Opening TLS connection to `%s'...%s"
 	     host (if done "done" "failed"))
     (when use-temp-buffer


João
-- 
João Távora

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 23:07:30 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> Here is a patch that should be equivalent to the defadvice I'm using.
JT> As I said, it works for me. Also I didn't have a VC copy of emacs so I
JT> used  `diff-buffer-with-file'

I applied something very similar.  It's a tiny change so it doesn't
require assignment papers.  If you'd like to contribute to GNU Emacs or
other GNU software in the future, I encourage you to file them.

revno: 112858
timestamp: Wed 2013-06-05 11:03:37 -0400
message:
  (open-tls-stream): Remove unneeded buffer contents when opening the connection.

This change didn't break anything in my testing.  Please test so I can
close the bug if it's fixed for you.

Ted

diff -u -L /usr/local/share/emacs/24.3/lisp/net/tls.el.gz -L
\#\<buffer\ tls.el.gz\> /tmp/jka-com1909LVh
/tmp/buffer-content-1909lpt
--- /usr/local/share/emacs/24.3/lisp/net/tls.el.gz
+++ #<buffer tls.el.gz>
@@ -286,7 +286,11 @@
			     (format "Host name in certificate doesn't \
 match `%s'. Connect anyway? " host))))))
	(setq done nil)
-	(delete-process process)))
+	(delete-process process))
+      ;; delete all the informational messages that could
+      ;; confuse futures users of `buffer'
+      ;;
+      (delete-region (point-min) (point)))
     (message "Opening TLS connection to `%s'...%s"
	     host (if done "done" "failed"))
     (when use-temp-buffer

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Glenn Morris]

Ted Zlatanov wrote:

> I applied something very similar.  It's a tiny change so it doesn't
> require assignment papers.

Thanks for applying it. If committing changes by others, please make the
ChangeLog entry in their name, and use commit --author.
And because this has a bug number, the ChangeLog entry can use that
rather than an emacs-devel link.

(Seems he already has an assignment BTW).

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Wed, 05 Jun 2013 12:42:44 -0400 Glenn Morris <rgm@gnu.org> wrote: 

GM> Ted Zlatanov wrote:
>> I applied something very similar.  It's a tiny change so it doesn't
>> require assignment papers.

GM> Thanks for applying it. If committing changes by others, please make the
GM> ChangeLog entry in their name, and use commit --author.

This must be a recent policy, or at least wasn't requested before.  Sorry.

GM> And because this has a bug number, the ChangeLog entry can use that
GM> rather than an emacs-devel link.

Right.  Can you tell I haven't contributed in a while? :)

Should I edit the ChangeLog with the bug number or just leave it alone
now that it's wrong?

Ted

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Glenn Morris]

Ted Zlatanov wrote:

> GM> Thanks for applying it. If committing changes by others, please make the
> GM> ChangeLog entry in their name, and use commit --author.
>
> This must be a recent policy, or at least wasn't requested before.  Sorry.

The ChangeLog thing is very old. The --commit thing is less well
advertised, but must be several years old by now I should think.

> Should I edit the ChangeLog with the bug number or just leave it alone
> now that it's wrong?

It's always fine to improve ChangeLog entries after the fact.
(I changed it.)

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

Thanks!

To test, I will load just load the trunks "tls.el" in my existing W32
emacs. (I don't have time to get a windows emacs-building toolchain).

As Stefan mentioned, I have already signed papers.

João

On Wed, Jun 5, 2013 at 4:06 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Mon, 20 May 2013 23:07:30 +0100 João Távora <joaotavora@gmail.com> wrote:
>
> JT> Here is a patch that should be equivalent to the defadvice I'm using.
> JT> As I said, it works for me. Also I didn't have a VC copy of emacs so I
> JT> used  `diff-buffer-with-file'
>
> I applied something very similar.  It's a tiny change so it doesn't
> require assignment papers.  If you'd like to contribute to GNU Emacs or
> other GNU software in the future, I encourage you to file them.
>
> revno: 112858
> timestamp: Wed 2013-06-05 11:03:37 -0400
> message:
>   (open-tls-stream): Remove unneeded buffer contents when opening the connection.
>
> This change didn't break anything in my testing.  Please test so I can
> close the bug if it's fixed for you.
>
> Ted
>
> diff -u -L /usr/local/share/emacs/24.3/lisp/net/tls.el.gz -L
> \#\<buffer\ tls.el.gz\> /tmp/jka-com1909LVh
> /tmp/buffer-content-1909lpt
> --- /usr/local/share/emacs/24.3/lisp/net/tls.el.gz
> +++ #<buffer tls.el.gz>
> @@ -286,7 +286,11 @@
>                              (format "Host name in certificate doesn't \
>  match `%s'. Connect anyway? " host))))))
>         (setq done nil)
> -       (delete-process process)))
> +       (delete-process process))
> +      ;; delete all the informational messages that could
> +      ;; confuse futures users of `buffer'
> +      ;;
> +      (delete-region (point-min) (point)))
>      (message "Opening TLS connection to `%s'...%s"
>              host (if done "done" "failed"))
>      (when use-temp-buffer
>



-- 
João Távora

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

Thanks!

To test, I will load just load the trunks "tls.el" in my existing W32
emacs. (I don't have time to get a windows emacs-building toolchain).

As Stefan mentioned, I have already signed papers.

João

On Wed, Jun 5, 2013 at 4:06 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Mon, 20 May 2013 23:07:30 +0100 João Távora <joaotavora@gmail.com> wrote:
>
> JT> Here is a patch that should be equivalent to the defadvice I'm using.
> JT> As I said, it works for me. Also I didn't have a VC copy of emacs so I
> JT> used  `diff-buffer-with-file'
>
> I applied something very similar.  It's a tiny change so it doesn't
> require assignment papers.  If you'd like to contribute to GNU Emacs or
> other GNU software in the future, I encourage you to file them.
>
> revno: 112858
> timestamp: Wed 2013-06-05 11:03:37 -0400
> message:
>   (open-tls-stream): Remove unneeded buffer contents when opening the connection.
>
> This change didn't break anything in my testing.  Please test so I can
> close the bug if it's fixed for you.
>
> Ted
>
> diff -u -L /usr/local/share/emacs/24.3/lisp/net/tls.el.gz -L
> \#\<buffer\ tls.el.gz\> /tmp/jka-com1909LVh
> /tmp/buffer-content-1909lpt
> --- /usr/local/share/emacs/24.3/lisp/net/tls.el.gz
> +++ #<buffer tls.el.gz>
> @@ -286,7 +286,11 @@
>                              (format "Host name in certificate doesn't \
>  match `%s'. Connect anyway? " host))))))
>         (setq done nil)
> -       (delete-process process)))
> +       (delete-process process))
> +      ;; delete all the informational messages that could
> +      ;; confuse futures users of `buffer'
> +      ;;
> +      (delete-region (point-min) (point)))
>      (message "Opening TLS connection to `%s'...%s"
>              host (if done "done" "failed"))
>      (when use-temp-buffer
>



-- 
João Távora

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 23:07:30 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> Here is a patch that should be equivalent to the defadvice I'm using.
JT> As I said, it works for me. Also I didn't have a VC copy of emacs so I
JT> used  `diff-buffer-with-file'

I applied something very similar.  It's a tiny change so it doesn't
require assignment papers.  If you'd like to contribute to GNU Emacs or
other GNU software in the future, I encourage you to file them.

revno: 112858
timestamp: Wed 2013-06-05 11:03:37 -0400
message:
  (open-tls-stream): Remove unneeded buffer contents when opening the connection.

This change didn't break anything in my testing.  Please test so I can
close the bug if it's fixed for you.

Ted

diff -u -L /usr/local/share/emacs/24.3/lisp/net/tls.el.gz -L
\#\<buffer\ tls.el.gz\> /tmp/jka-com1909LVh
/tmp/buffer-content-1909lpt
--- /usr/local/share/emacs/24.3/lisp/net/tls.el.gz
+++ #<buffer tls.el.gz>
@@ -286,7 +286,11 @@
			     (format "Host name in certificate doesn't \
 match `%s'. Connect anyway? " host))))))
	(setq done nil)
-	(delete-process process)))
+	(delete-process process))
+      ;; delete all the informational messages that could
+      ;; confuse futures users of `buffer'
+      ;;
+      (delete-region (point-min) (point)))
     (message "Opening TLS connection to `%s'...%s"
	     host (if done "done" "failed"))
     (when use-temp-buffer

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

On Mon, May 20, 2013 at 12:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>
> Using an external binary to transport SSL or TLS is a hack IMO.

I see, and reading a bit of tls.el it makes sense. It does all this
crazy regexp searching... This is a hack in my book too, albeit one
that's been working fine,

> Did you propose a patch?  I would commit a patch but can't write it
> despite your great description of the problem.
>

Here is a patch that should be equivalent to the defadvice I'm using.
As I said, it works for me. Also I didn't have a VC copy of emacs so I
used  `diff-buffer-with-file'

diff -u -L /usr/local/share/emacs/24.3/lisp/net/tls.el.gz -L
\#\<buffer\ tls.el.gz\> /tmp/jka-com1909LVh
/tmp/buffer-content-1909lpt
--- /usr/local/share/emacs/24.3/lisp/net/tls.el.gz
+++ #<buffer tls.el.gz>
@@ -286,7 +286,11 @@
 			     (format "Host name in certificate doesn't \
 match `%s'. Connect anyway? " host))))))
 	(setq done nil)
-	(delete-process process)))
+	(delete-process process))
+      ;; delete all the informational messages that could
+      ;; confuse futures users of `buffer'
+      ;;
+      (delete-region (point-min) (point)))
     (message "Opening TLS connection to `%s'...%s"
 	     host (if done "done" "failed"))
     (when use-temp-buffer


João
-- 
João Távora

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

[I removed emacs-devel.]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: 14380@debbugs.gnu.org, emacs-devel@gnu.org
> Date: Sat, 18 May 2013 23:17:02 -0400
> 
> (CC to emacs-devel as I think this discussion is relevant there)

There's no reason: all the people who you'd like to reach read the bug
list as well.

> The big problem for me is that I don't have the time or platform
> knowledge to write a GnuTLS auto-installer and updater for those two
> problematic platforms.  The GnuTLS developers don't want to provide this
> service either.  Who will be responsible to it?  What happens when a
> security vulnerability hits the DLLs we distribute with Emacs?
> 
> My proposal would be to push out the next Emacs bundled with the latest
> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
> updating them, and treat GnuTLS vulnerabilities as Emacs
> vulnerabilities.  This is not ideal but IMO better than the current
> situation.

I see no problems with the current situation.  Installing precompiled
GnuTLS from a zip file is a snap.

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sun, 19 May 2013 18:32:37 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> My proposal would be to push out the next Emacs bundled with the latest
>> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
>> updating them, and treat GnuTLS vulnerabilities as Emacs
>> vulnerabilities.  This is not ideal but IMO better than the current
>> situation.

EZ> I see no problems with the current situation.  Installing precompiled
EZ> GnuTLS from a zip file is a snap.

That's only a small part of the risk and responsibility we're shifting
onto the Emacs users.

Ted

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: 14380@debbugs.gnu.org,  joaotavora@gmail.com
> Date: Mon, 20 May 2013 09:56:27 -0400
> 
> On Sun, 19 May 2013 18:32:37 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
> 
> >> My proposal would be to push out the next Emacs bundled with the latest
> >> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
> >> updating them, and treat GnuTLS vulnerabilities as Emacs
> >> vulnerabilities.  This is not ideal but IMO better than the current
> >> situation.
> 
> EZ> I see no problems with the current situation.  Installing precompiled
> EZ> GnuTLS from a zip file is a snap.
> 
> That's only a small part of the risk and responsibility we're shifting
> onto the Emacs users.

What risk? what responsibility?

A user who installs software on her computer is already trusted with
certain responsibilities, because a single mistyped command or a badly
built package can easily shut down a perfectly healthy system for
hours, if not days.  Users install dozens of packages needed to create
a workable environment for whatever they need to accomplish.  Why is
GnuTLS so special?

And mind you, in view of the latest sparring between GnuTLS developers
and the FSF (which I have no idea how ended, except that the license
was downgraded a bit and the official site moved), I'm not even sure
the FSF will agree to distribute GnuTLS with Emacs, on any platform.
Why should Emacs development enter this minefield?  And for what? for
solving a non-existing problem of installing a simple package?  Don't
we have better places to apply our time and energy?

Don't misunderstand me: if someone decides to provide regular builds
of GnuTLS ready to be downloaded and installed, I will applaud that
person.  Heck, it will be one less duty for me, for starters, as far
as the Windows binaries are concerned.  But please don't represent
this as a must for Emacs, because it isn't.

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 19:28:40 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Cc: 14380@debbugs.gnu.org,  joaotavora@gmail.com
>> Date: Mon, 20 May 2013 09:56:27 -0400
>> 
>> On Sun, 19 May 2013 18:32:37 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
>> 
>> >> My proposal would be to push out the next Emacs bundled with the latest
>> >> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
>> >> updating them, and treat GnuTLS vulnerabilities as Emacs
>> >> vulnerabilities.  This is not ideal but IMO better than the current
>> >> situation.
>> 
EZ> I see no problems with the current situation.  Installing precompiled
EZ> GnuTLS from a zip file is a snap.
>> 
>> That's only a small part of the risk and responsibility we're shifting
>> onto the Emacs users.

EZ> What risk? what responsibility?

The risk is that their version of GnuTLS is out of date.  The
responsibility is to update it regularly.

EZ> A user who installs software on her computer is already trusted with
EZ> certain responsibilities, because a single mistyped command or a badly
EZ> built package can easily shut down a perfectly healthy system for
EZ> hours, if not days.  Users install dozens of packages needed to create
EZ> a workable environment for whatever they need to accomplish.  Why is
EZ> GnuTLS so special?

Installing and keeping GnuTLS up to date should not be the
responsibility of the user.

To put it another way, if you want that responsibility, you're in a very
small percentage of the Emacs user population.  Most users don't want it
and will neglect it badly.

EZ> And mind you, in view of the latest sparring between GnuTLS developers
EZ> and the FSF (which I have no idea how ended, except that the license
EZ> was downgraded a bit and the official site moved), I'm not even sure
EZ> the FSF will agree to distribute GnuTLS with Emacs, on any platform.
EZ> Why should Emacs development enter this minefield?

That's a reasonable question.  I think we have to face it regardless of
the outcome of this discussion because Emacs depends on GnuTLS for SSL
and TLS communications right now.

As far as I know GnuTLS status is back to "kosher."

EZ> And for what? for solving a non-existing problem of installing a
EZ> simple package?

Installing is easy.  Keeping it up to date isn't.  Security updates are
tedious and tedious things get overlooked.

EZ> Don't misunderstand me: if someone decides to provide regular builds
EZ> of GnuTLS ready to be downloaded and installed, I will applaud that
EZ> person.  Heck, it will be one less duty for me, for starters, as far
EZ> as the Windows binaries are concerned.  But please don't represent
EZ> this as a must for Emacs, because it isn't.

I see it as a responsibility we're avoiding.  But if we had these
regular builds, how would the user know about a critical update he
really must install?

See here http://bugs.python.org/issue17425 for an example of how the
Python community dealt with an security issue in the OpenSSL libraries
they ship for Windows.  I guess we have to answer the question of
whether that's a standard we as Emacs developers should aspire to, or
not.

Ted

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: 14380@debbugs.gnu.org,  joaotavora@gmail.com
> Date: Fri, 24 May 2013 15:48:20 -0400
> 
> EZ> What risk? what responsibility?
> 
> The risk is that their version of GnuTLS is out of date.

That happens with dozens of packages on each user's machine.  There's
nothing in GnuTLS that makes it unique in this regard.

Moreover, the latest and greatest GnuTLS sometimes simply won't build
on some systems.  Like with the latest release, for example.  How is
it a good idea to upgrade to a version that is broken?  And if the
latest version is not always the one to upgrade to, then who will make
the research required to tell users to which version to upgrade?  You?

I did that research for the single version whose Windows port I made
available.  I built it, fixed the build problems, tested it, fixed the
problems revealed by that, and after doing all that I could in good
faith tell people they can use that version without too much fear.
Why is it safer for users to upgrade to a newer version than to stay
with the one I tested?  Shouldn't whoever wants to tell them to
upgrade invest a similar effort in that newer version?  If she
doesn't, she is actually shifting the responsibility to the users
anyway!

> The responsibility is to update it regularly.

Or not.  Blindly upgrading could get users in trouble.

> EZ> A user who installs software on her computer is already trusted with
> EZ> certain responsibilities, because a single mistyped command or a badly
> EZ> built package can easily shut down a perfectly healthy system for
> EZ> hours, if not days.  Users install dozens of packages needed to create
> EZ> a workable environment for whatever they need to accomplish.  Why is
> EZ> GnuTLS so special?
> 
> Installing and keeping GnuTLS up to date should not be the
> responsibility of the user.

Says you.  But since there's no one else to pick up the gauntlet,
that's where this responsibility will need to rest.  If J.R. Hacker
needs GnuTLS today, he has no one else but himself to rely on.  All
we, the Emacs developers, do is just talk.

> To put it another way, if you want that responsibility, you're in a very
> small percentage of the Emacs user population.  Most users don't want it
> and will neglect it badly.

Again, nothing new or special here.

> As far as I know GnuTLS status is back to "kosher."

Not sure based on what you say this.

> I see it as a responsibility we're avoiding.  But if we had these
> regular builds, how would the user know about a critical update he
> really must install?
> 
> See here http://bugs.python.org/issue17425 for an example of how the
> Python community dealt with an security issue in the OpenSSL libraries
> they ship for Windows.  I guess we have to answer the question of
> whether that's a standard we as Emacs developers should aspire to, or
> not.

I'm sorry, but you are expecting from the Emacs development something
it can never provide in its present shape and form.  Tracking security
issues to this degree in even a single package is a very time
consuming job.  Unless we have several volunteers on board taking
responsibility for the various packages which Emacs supports, what you
seem to want is nothing more than a pipe dream.  I don't see any such
volunteers; in fact, I don't even see a single one.  If we had such an
individual, my year-old port would have been replaced by newer ones
already.  (Of course, the Windows build in GnuTLS is regularly broken,
so it's not really easy, either.)  Until that changes, all this talk
is just a huge waste of energy.  

If you think this kind of effort is possible, how about if you present
a complete realistic plan for having a secure Emacs, name individuals
who would test the releases of those packages for security issues, and
make sure any problems that are detected are promptly fixed on all
platforms we support, etc.?  Otherwise, let's just stop these endless
discussions and admit that we don't have the resources to live up to
it.

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

Here's a emacs -Q recipe:

    $ emacs -Q
    M-x eval-expression RET (setq gnus-select-method '(nnimap "imaps.mycompany.com")) RET
    M-x gnus

An error pops up:

    nnimap (imaps.mycompany.com) open error: ''gnutls-cli' is not recognized as an internal or external command,

In the *Message* buffer, we can see at at least one tls command succeeded

    Opening connection to imaps.mycompany.com via tls...
    gnutls.c: [1] (Emacs) GnuTLS library not found
    Opening TLS connection to `imaps.mycompany.com'...
    Opening TLS connection with `gnutls-cli --insecure -p 993 imaps.mycompany.com'...failed
    Opening TLS connection with `gnutls-cli --insecure -p 993 imaps.mycompany.com --protocols ssl3'...failed
    Opening TLS connection with `openssl s_client -connect imaps.mycompany.com:993 -no_ssl2 -ign_eof'...done
    Opening TLS connection to `imaps.mycompany.com'...done
    nnimap (imaps.mycompany.com) open error: ''gnutls-cli' is not recognized as an internal or external command,
    '.  Continue? (y or n)  n

This works fine, on the same server with linux.

My analysis and possible fix is as follows: The problem is that
`network-stream-open-tls' returns invalid information in its final form:

    (list stream
          (network-stream-get-response stream start eoc)
          (network-stream-command stream capability-command eoc)
          'tls)

This is turn is because code immediately preceding it

    ;; If we're using tls.el, we have to delete the output from
    ;; openssl/gnutls-cli.
    (when (and (null use-builtin-gnutls)
               eoc)
      (network-stream-get-response stream start eoc)
      (goto-char (point-min))
      (when (re-search-forward eoc nil t)
        (goto-char (match-beginning 0))
        (delete-region (point-min) (line-beginning-position))))

is not really doing what it says it's doing: it's not deleting enough
input, and the search for capabilities actually returns the IMAP
greeting.

This in turn is related to the CRLF lineendings that the function
`open-tls-stream' running under w32 only (in linux it doesn't happen)
inserts in the process buffer before the end of the tls handshake. The
code in `network-stream-open-tls' for getting rid of useless starting
output deletes up to the first CRLF (and apparently it should do so just
once, since other CRLF's are legitimate). But if fails and catches a
very early CRLF which is still part of the now useless TLS
handshake. The search performed by

    (network-stream-command stream capability-command eoc)

doesn't work. That said, a reasonable way to fix this might be adding a
line

    (delete-region (point-min) (point))

as the first form in the body of the `when'. This is because
`open-tls-stream' apparently guarantees that point will be left where
the tls handshake ended, as per its code snippet and comment:

    (if start-of-data
        ;; move point to start of client data
        (goto-char start-of-data))

Alternalively, make `open-tls-stream' do that itself in the end. I'm
doing just that with.

    (defadvice open-tls-stream (after dont-forget-to-erase-buffer-up-to-point activate)
      (delete-region (point-min) (point)))

Or maybe improve searches by `network-stream-get-response' or be smarter
when deleting output.

Here's the contents of the nnimap process buffer. Notice the CRLFs.


'gnutls-cli' is not recognized as an internal or external command,
operable program or batch file.

Process *nnimap* exited abnormally with code 1
'gnutls-cli' is not recognized as an internal or external command,
operable program or batch file.

Process *nnimap* exited abnormally with code 1
depth=0 C = PT, L = Lisboa, O = SISCOG.COM, OU = IMAP server, CN = *.siscog.com, emailAddress = suporte@siscog.pt
verify error:num=18:self signed certificate
verify return:1
depth=0 C = PT, L = Lisboa, O = SISCOG.COM, OU = IMAP server, CN = *.siscog.com, emailAddress = suporte@siscog.pt
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP server/CN=*.siscog.com/emailAddress=suporte@siscog.pt
   i:/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP server/CN=*.siscog.com/emailAddress=suporte@siscog.pt
---
Server certificate
-----BEGIN CERTIFICATE-----
[elided]
-----END CERTIFICATE-----
subject=/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP server/CN=*.siscog.com/emailAddress=suporte@siscog.pt
issuer=/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP server/CN=*.siscog.com/emailAddress=suporte@siscog.pt
---
No client certificate CA names sent
---
SSL handshake has read 1240 bytes and written 518 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 9334E31E9FD9A90C9D301227A2799EB8210AAC9FFD583B6597251744CB6EBFB3
    Session-ID-ctx:
    Master-Key: 7A5A40FEFC3A3493C9AFC4B80BCB044FAC48C94E0B4DE6E35035866F90EC78FFDE18D1B40E93B125F4F3F73468A9D5F0
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Compression: 1 (zlib compression)
    Start Time: 1368189548
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---
* OK Dovecot ready.

In GNU Emacs 24.3.1 (i386-mingw-nt5.1.2600)
 of 2013-03-17 on MARVIN
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
 `configure --with-gcc (4.7) --cflags
 -ID:/devel/emacs/libs/libXpm-3.5.8/include
 -ID:/devel/emacs/libs/libXpm-3.5.8/src
 -ID:/devel/emacs/libs/libpng-dev_1.4.3-1/include
 -ID:/devel/emacs/libs/zlib-dev_1.2.5-2/include
 -ID:/devel/emacs/libs/giflib-4.1.4-1/include
 -ID:/devel/emacs/libs/jpeg-6b-4/include
 -ID:/devel/emacs/libs/tiff-3.8.2-1/include
 -ID:/devel/emacs/libs/gnutls-3.0.9/include
 -ID:/devel/emacs/libs/libiconv-1.13.1-1-dev/include
 -ID:/devel/emacs/libs/libxml2-2.7.8/include/libxml2'

Important settings:
  value of $LANG: C.UTF-8
  locale-coding-system: cp1252
  default enable-multibyte-characters: t

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: joaotavora@gmail.com (João Távora)
> Date: Fri, 10 May 2013 13:49:20 +0100
> 
> 
>     (list stream
>           (network-stream-get-response stream start eoc)
>           (network-stream-command stream capability-command eoc)
>           'tls)
> 
> This is turn is because code immediately preceding it
> 
>     ;; If we're using tls.el, we have to delete the output from
>     ;; openssl/gnutls-cli.
>     (when (and (null use-builtin-gnutls)
>                eoc)
>       (network-stream-get-response stream start eoc)
>       (goto-char (point-min))
>       (when (re-search-forward eoc nil t)
>         (goto-char (match-beginning 0))
>         (delete-region (point-min) (line-beginning-position))))
> 
> is not really doing what it says it's doing: it's not deleting enough
> input, and the search for capabilities actually returns the IMAP
> greeting.

Why don't you use the built-in gnutls?  This is the recommended way of
using GnuTLS on Windows.

Or did I misunderstand your analysis?

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

More or less, you didn't misunderstand my analysis, but I didn't use
built-in tls because its not the default (I did see some
references to it, and the function seems to be fboundp (whereas in linux
its not) but the code path didn't lead that way for some reason).

With emacs -Q, setting gnus-select-method that way and then doing M-x gnus
fails.

If the builtin tls fixes this and its the "recommended way" it should
be the default no?

Maybe it *is* the default but emacs -Q possibly configured itself to use
some external gnutls and configure itself after it. I'm using cygwin, so
that might be the case. In that case, this might be seen as a cygwin bug,
in which case I apologize for the noise.

btw: how to I configure emacs to use builtin gnutls?

J

On Fri, May 10, 2013 at 3:00 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: joaotavora@gmail.com (João Távora)
> > Date: Fri, 10 May 2013 13:49:20 +0100
> >
> >
> >     (list stream
> >           (network-stream-get-response stream start eoc)
> >           (network-stream-command stream capability-command eoc)
> >           'tls)
> >
> > This is turn is because code immediately preceding it
> >
> >     ;; If we're using tls.el, we have to delete the output from
> >     ;; openssl/gnutls-cli.
> >     (when (and (null use-builtin-gnutls)
> >                eoc)
> >       (network-stream-get-response stream start eoc)
> >       (goto-char (point-min))
> >       (when (re-search-forward eoc nil t)
> >         (goto-char (match-beginning 0))
> >         (delete-region (point-min) (line-beginning-position))))
> >
> > is not really doing what it says it's doing: it's not deleting enough
> > input, and the search for capabilities actually returns the IMAP
> > greeting.
>
> Why don't you use the built-in gnutls?  This is the recommended way of
> using GnuTLS on Windows.
>
> Or did I misunderstand your analysis?




--
João Távora

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 10 May 2013 17:00:32 +0100
> Cc: 14380@debbugs.gnu.org
> 
> If the builtin tls fixes this and its the "recommended way" it should
> be the default no?

It _is_ the default, AFAIK, but you need to install the GnuTLS
libraries (DLLs) for it to work.  Try installing this one:

  http://sourceforge.net/projects/ezwinports/files/gnutls-3.0.9-w32-bin.zip/download

> btw: how to I configure emacs to use builtin gnutls?

I think you just need to install the above (and restart Emacs, if it's
already running).

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

OK, I see.

Honestly I thought builtin tls was some kind of tls library in elisp.
Needing external libraries doesn't make it very builtin :-), but
that's just my opinion.

In any case, my description shows how it "almost worked" with whatever
tls client it found (it was probably cygwin's, but I didn't check).
You might want to avoid these bug reports by fixing it the way I
suggested or some other way. Or not. All is OK from my part.

Thanks for the support,
João

On Fri, May 10, 2013 at 6:17 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: João Távora <joaotavora@gmail.com>
>> Date: Fri, 10 May 2013 17:00:32 +0100
>> Cc: 14380@debbugs.gnu.org
>>
>> If the builtin tls fixes this and its the "recommended way" it should
>> be the default no?
>
> It _is_ the default, AFAIK, but you need to install the GnuTLS
> libraries (DLLs) for it to work.  Try installing this one:
>
>   http://sourceforge.net/projects/ezwinports/files/gnutls-3.0.9-w32-bin.zip/download
>
>> btw: how to I configure emacs to use builtin gnutls?
>
> I think you just need to install the above (and restart Emacs, if it's
> already running).



-- 
João Távora

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 10 May 2013 21:44:12 +0100
> Cc: 14380@debbugs.gnu.org
> 
> OK, I see.
> 
> Honestly I thought builtin tls was some kind of tls library in elisp.
> Needing external libraries doesn't make it very builtin :-), but
> that's just my opinion.

It's builtin in the sense that it does not need external programs.

> In any case, my description shows how it "almost worked" with whatever
> tls client it found (it was probably cygwin's, but I didn't check).
> You might want to avoid these bug reports by fixing it the way I
> suggested or some other way. Or not. All is OK from my part.

The native Windows build of Emacs doesn't work very well with an
external tls client, because their interaction is through signals,
which are not supported by Windows.

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Fri, 10 May 2013 21:44:12 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> Honestly I thought builtin tls was some kind of tls library in elisp.
JT> Needing external libraries doesn't make it very builtin :-), but
JT> that's just my opinion.

Implementing TLS in ELisp is possible, sure, but it would be extremely
slow and there's considerable risk in reimplementing that protocol.  We
chose to use GnuTLS instead, which is a fast reliable C implementation
and is available for W32 and all our other platforms.

JT> In any case, my description shows how it "almost worked" with whatever
JT> tls client it found (it was probably cygwin's, but I didn't check).
JT> You might want to avoid these bug reports by fixing it the way I
JT> suggested or some other way. Or not. All is OK from my part.

I've seen dozens of bugs related to "almost working" external TLS
binaries on all platforms.  It was one of the reasons I did the initial
GnuTLS integration with Emacs.  My vote is to require GnuTLS with Emacs
and to only support it, but there are some questions there, mainly for
W32 and Mac OS X: do we auto-update GnuTLS?  What happens when the
GnuTLS we install conflicts with another system install?  And so on...

Ted

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Fri, 24 May 2013 23:27:07 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Cc: 14380@debbugs.gnu.org,  joaotavora@gmail.com
>> Date: Fri, 24 May 2013 15:48:20 -0400
>> 
EZ> What risk? what responsibility?
>> 
>> The risk is that their version of GnuTLS is out of date.

EZ> That happens with dozens of packages on each user's machine.  There's
EZ> nothing in GnuTLS that makes it unique in this regard.

Yes, of course.  I don't know the other packages we require to enable
extra features on W32, sorry.  I think GnuTLS is somewhat unique in this
regard by being the only way to do secure communications with the
outside world, but it's worth considering putting the other packages
under the same mechanism as GnuTLS for installations and updates.

EZ> Moreover, the latest and greatest GnuTLS sometimes simply won't build
EZ> on some systems.  Like with the latest release, for example.  How is
EZ> it a good idea to upgrade to a version that is broken?  And if the
EZ> latest version is not always the one to upgrade to, then who will make
EZ> the research required to tell users to which version to upgrade?
EZ> You?

Yes, possibly.

EZ> I did that research for the single version whose Windows port I made
EZ> available.  I built it, fixed the build problems, tested it, fixed the
EZ> problems revealed by that, and after doing all that I could in good
EZ> faith tell people they can use that version without too much fear.
EZ> Why is it safer for users to upgrade to a newer version than to stay
EZ> with the one I tested?  Shouldn't whoever wants to tell them to
EZ> upgrade invest a similar effort in that newer version?  If she
EZ> doesn't, she is actually shifting the responsibility to the users
EZ> anyway!

You're right.  It's a lot of work.  I appreciate it very much.  I hope
to be able to find the resources to make these reviews happen.

>> Installing and keeping GnuTLS up to date should not be the
>> responsibility of the user.

EZ> Says you.  But since there's no one else to pick up the gauntlet,
EZ> that's where this responsibility will need to rest.  If J.R. Hacker
EZ> needs GnuTLS today, he has no one else but himself to rely on.  All
EZ> we, the Emacs developers, do is just talk.

I like to ask before I make changes, hence my request for votes in
emacs-devel.  Sorry if it seems like empty talk to you.

>> As far as I know GnuTLS status is back to "kosher."

EZ> Not sure based on what you say this.

Monitoring the GnuTLS mailing lists.  I don't mean there are no issues,
only that the FSF has not made a statement about changing its preference
for GnuTLS.

>> I see it as a responsibility we're avoiding.  But if we had these
>> regular builds, how would the user know about a critical update he
>> really must install?
>> 
>> See here http://bugs.python.org/issue17425 for an example of how the
>> Python community dealt with an security issue in the OpenSSL libraries
>> they ship for Windows.  I guess we have to answer the question of
>> whether that's a standard we as Emacs developers should aspire to, or
>> not.

EZ> I'm sorry, but you are expecting from the Emacs development something
EZ> it can never provide in its present shape and form.  Tracking security
EZ> issues to this degree in even a single package is a very time
EZ> consuming job.  Unless we have several volunteers on board taking
EZ> responsibility for the various packages which Emacs supports, what you
EZ> seem to want is nothing more than a pipe dream.  I don't see any such
EZ> volunteers; in fact, I don't even see a single one.  If we had such an
EZ> individual, my year-old port would have been replaced by newer ones
EZ> already.  (Of course, the Windows build in GnuTLS is regularly broken,
EZ> so it's not really easy, either.)  Until that changes, all this talk
EZ> is just a huge waste of energy.

EZ> If you think this kind of effort is possible, how about if you present
EZ> a complete realistic plan for having a secure Emacs, name individuals
EZ> who would test the releases of those packages for security issues, and
EZ> make sure any problems that are detected are promptly fixed on all
EZ> platforms we support, etc.?  Otherwise, let's just stop these endless
EZ> discussions and admit that we don't have the resources to live up to
EZ> it.

I'm trying to get the work started by first and foremost deciding if
Emacs as a project wants to do this at all.  This is a decision for the
maintainers and you've voted against it on emacs-devel, so let's see
what the vote count is and what the maintainers say.

If the maintainers are OK with this direction, I will start working on
automating the builds (which will need your help initially, if you're
willing, to replicate your build process), asking for volunteers, and
packaging the libraries.  I don't have a 5-year plan but hope to get far
enough to make it something sustainable.

Ted

bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 24 May 2013 18:20:55 -0400
> 
> >> The risk is that their version of GnuTLS is out of date.
> 
> EZ> That happens with dozens of packages on each user's machine.  There's
> EZ> nothing in GnuTLS that makes it unique in this regard.
> 
> Yes, of course.  I don't know the other packages we require to enable
> extra features on W32, sorry.

Those are the image libraries and libxml2.  And, of course, any
prerequisite packages needed to build those: zlib, libintl, libiconv,
and the libraries, like libnettle, that GnuTLS depends on.

> I think GnuTLS is somewhat unique in this regard by being the only
> way to do secure communications with the outside world, but it's
> worth considering putting the other packages under the same
> mechanism as GnuTLS for installations and updates.

Security is not just about secure communications.  I'm sure you know
that.

> >> Installing and keeping GnuTLS up to date should not be the
> >> responsibility of the user.
> 
> EZ> Says you.  But since there's no one else to pick up the gauntlet,
> EZ> that's where this responsibility will need to rest.  If J.R. Hacker
> EZ> needs GnuTLS today, he has no one else but himself to rely on.  All
> EZ> we, the Emacs developers, do is just talk.
> 
> I like to ask before I make changes, hence my request for votes in
> emacs-devel.  Sorry if it seems like empty talk to you.

It is empty because it doesn't lead to any action.  We've been talking
about this for months already.

> >> As far as I know GnuTLS status is back to "kosher."
> 
> EZ> Not sure based on what you say this.
> 
> Monitoring the GnuTLS mailing lists.

The actual discussions were private, not on any public list.

> I'm trying to get the work started by first and foremost deciding if
> Emacs as a project wants to do this at all.  This is a decision for the
> maintainers and you've voted against it on emacs-devel, so let's see
> what the vote count is and what the maintainers say.

I think it's well beyond our capacities and resources, yes.