Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

(CC to emacs-devel as I think this discussion is relevant there)

On Sat, 18 May 2013 14:05:47 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> The point [is] that needing external libraries which are not always
JT> bundled doesn't make it very "builtin".

I'm not bringing GnuTLS into the Emacs source tree, which is the only
other way to make it built-in functionality.  I understand there are
issues with external dependencies and in fact I asked that we bundle the
GnuTLS W32 DLLs with the W32 Emacs builds.  That led to a long
discussions about how that makes security our responsibility and how we
then need to deal with GnuTLS updates, and I didn't have a strong desire
to become a W32 distribution expert since I barely know that platform.
No one else picked it up, and there we are with "install it yourself" as
the recommended way to get GnuTLS to work on W32.

>> I've seen dozens of bugs related to "almost working" external TLS
>> binaries on all platforms.

JT> Yes, but have you looked closely at this particular one? The point is rather
JT> to increase robustness. That is, `open-tls-stream` could/should promise
JT> to cleanup the process buffer of its handshake garbage, so that future
JT> functions that use that resource don't see it and don't get confused by it.

JT> I'm assuming they don't need to see it, I might be wrong.

I'm not able to fix this bug or work on bugs in the external SSL/TLS support.

JT> But if I'm right and that fix is performed then you've effectively extended
JT> "imap just works" the set of W32 emacs users who type "M-x gnus" on a
JT> vanilla emacs in a system with some cygwin installation in PATH. Maybe it's
JT> a small set but I'm in it (when I'm at work).

Wouldn't you rather get GnuTLS to work by default?  Otherwise we serve
the use case "I have no secure transport, so let me use a hack by
default."

>> GnuTLS integration with Emacs.  My vote is to require GnuTLS with Emacs
>> and to only support it, but there are some questions there, mainly for
>> W32 and Mac OS X: do we auto-update GnuTLS?  What happens when the
>> GnuTLS we install conflicts with another system install?  And so on...

JT> That's all fine, I guess. I vote for that too :-)

The big problem for me is that I don't have the time or platform
knowledge to write a GnuTLS auto-installer and updater for those two
problematic platforms.  The GnuTLS developers don't want to provide this
service either.  Who will be responsible to it?  What happens when a
security vulnerability hits the DLLs we distribute with Emacs?

My proposal would be to push out the next Emacs bundled with the latest
GnuTLS DLLs, only support GnuTLS, provide users with instructions on
updating them, and treat GnuTLS vulnerabilities as Emacs
vulnerabilities.  This is not ideal but IMO better than the current
situation.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

On Sun, May 19, 2013 at 4:17 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> GnuTLS W32 DLLs with the W32 Emacs builds.  That led to a long
> discussions about how that makes security our responsibility and how we

I see. Indeed, bundling security stuff with your app is increasing
its responsibility manifold.

> Wouldn't you rather get GnuTLS to work by default?  Otherwise we serve
> the use case "I have no secure transport, so let me use a hack by
> default."

I don't understand. What is the hack here? External binary for TLS?
But yes, GnuTLS by default is certainly better...

> service either.  Who will be responsible to it?  What happens when a
> security vulnerability hits the DLLs we distribute with Emacs?
>
> My proposal would be to push out the next Emacs bundled with the latest
> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
> updating them, and treat GnuTLS vulnerabilities as Emacs
> vulnerabilities.  This is not ideal but IMO better than the current
> situation.

... but then you have all these headaches.

The fix I proposed aims for the status quo, that is: make external
TLS binary support slightly more robust. My test case is even smaller:

* W32
* cygwin carrying the responsibility burden
* vanilla emacs working with tls/imap/gnus.

Thanks for the time spent in analysing this,
-- 
João Távora

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sun, 19 May 2013 12:45:12 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> On Sun, May 19, 2013 at 4:17 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Wouldn't you rather get GnuTLS to work by default?  Otherwise we serve
>> the use case "I have no secure transport, so let me use a hack by
>> default."

JT> I don't understand. What is the hack here? External binary for TLS?

Using an external binary to transport SSL or TLS is a hack IMO.

>> My proposal would be to push out the next Emacs bundled with the latest
>> GnuTLS DLLs, only support GnuTLS, provide users with instructions on
>> updating them, and treat GnuTLS vulnerabilities as Emacs
>> vulnerabilities.  This is not ideal but IMO better than the current
>> situation.

JT> ... but then you have all these headaches.

It's a headache I'm willing to endure for the sake of Emacs users.

The alternative, which João is enduring now, is to punt the problem.

This is a question for the Emacs maintainers: do you agree with me on
the above plan?  It would mean changing the way Mac OS X and W32 Emacs
builds are distributed, to include the GnuTLS libraries with the build,
and we'd have to implement a way (perhaps through the ELPA) to
distribute updates to these libraries.

JT> The fix I proposed aims for the status quo, that is: make external
JT> TLS binary support slightly more robust. My test case is even smaller:

JT> * W32
JT> * cygwin carrying the responsibility burden
JT> * vanilla emacs working with tls/imap/gnus.

Did you propose a patch?  I would commit a patch but can't write it
despite your great description of the problem.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Juanma Barranquero]

On Mon, May 20, 2013 at 1:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:

> It would mean changing the way Mac OS X and W32 Emacs
> builds are distributed, to include the GnuTLS libraries with the build,
> and we'd have to implement a way (perhaps through the ELPA) to
> distribute updates to these libraries.

Why would that be a good idea now if it wasn't seen as such before?

    J

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 04:08:57 +0200 Juanma Barranquero <lekktu@gmail.com> wrote: 

JB> On Mon, May 20, 2013 at 1:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> It would mean changing the way Mac OS X and W32 Emacs
>> builds are distributed, to include the GnuTLS libraries with the build,
>> and we'd have to implement a way (perhaps through the ELPA) to
>> distribute updates to these libraries.

JB> Why would that be a good idea now if it wasn't seen as such before?

Because we've had years of experience supporting GnuTLS, we're moving to
GnuTLS 3.x soon (there were no objections to my proposal for that), the
ELPA has matured and requires a secure transport[1], and we've heard
many times from our users on those two platforms that we should.

The current situation is worse: users are on their own and the GnuTLS
developers have not given them a way to auto-update or indicated
interest in doing so (I asked last year IIRC, and see
http://josefsson.org/gnutls4win/)

Ted

[1] I also proposed the ELPA as the way to distribute GnuTLS updates.
There's a separate thread about signing ELPA packages which would let us
avoid using a broken GnuTLS to pull its own updates, but that's not
essential to my proposal.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Juanma Barranquero]

On Mon, May 20, 2013 at 11:58 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:

> Because we've had years of experience supporting GnuTLS, we're moving to
> GnuTLS 3.x soon (there were no objections to my proposal for that), the
> ELPA has matured and requires a secure transport[1], and we've heard
> many times from our users on those two platforms that we should.

It still seems to me like a pretty serious change in our distribution
model, just because of GnuTLS on Windows and Mac OS.

I agree with Eli that installing GnuTLS from a zip file is easy enough.

   J

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 17:21:47 +0200 Juanma Barranquero <lekktu@gmail.com> wrote: 

JB> On Mon, May 20, 2013 at 11:58 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Because we've had years of experience supporting GnuTLS, we're moving to
>> GnuTLS 3.x soon (there were no objections to my proposal for that), the
>> ELPA has matured and requires a secure transport[1], and we've heard
>> many times from our users on those two platforms that we should.

JB> It still seems to me like a pretty serious change in our distribution
JB> model, just because of GnuTLS on Windows and Mac OS.

I think in the distribution of libraries on those two platforms we are
not even in the same millennium as most other software there.  I'm
willing to live with the non-security libraries like libxml (although
those can be compromised too).  But we should at least make an effort to
tell the user they are out of date, even if we do nothing more.

JB> I agree with Eli that installing GnuTLS from a zip file is easy enough.

OK, 2 votes against and 1 in favor.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 24 May 2013 15:32:40 -0400
> 
> I think in the distribution of libraries on those two platforms we are
> not even in the same millennium as most other software there.

You (or someone else) are welcome to take my ports and package them as
you see fit.  If you do a good job, I will even point to those
packages in nt/INSTALL etc.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Fri, 24 May 2013 22:50:01 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Fri, 24 May 2013 15:32:40 -0400
>> 
>> I think in the distribution of libraries on those two platforms we are
>> not even in the same millennium as most other software there.

EZ> You (or someone else) are welcome to take my ports and package them as
EZ> you see fit.  If you do a good job, I will even point to those
EZ> packages in nt/INSTALL etc.

Your work is very good and OK with me (although I can work with you to
automate the builds).  I am talking about distributing these binary
libraries better, especially critical updates.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 24 May 2013 17:55:16 -0400
> 
> EZ> You (or someone else) are welcome to take my ports and package them as
> EZ> you see fit.  If you do a good job, I will even point to those
> EZ> packages in nt/INSTALL etc.
> 
> Your work is very good and OK with me (although I can work with you to
> automate the builds).  I am talking about distributing these binary
> libraries better, especially critical updates.

That's exactly what I was talking about, too: take the port packaged
as drop-in zip files, and package it in some other way you consider to
be better.

As for critical updates, if you point me to the place or places where
these are described, I might consider whether it's time to update the
port.  (Please don't just point me to the gnutls-devel list, because I
read that, and have yet to see something like what you seem to allude
to.)

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sat, 25 May 2013 09:42:56 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Fri, 24 May 2013 17:55:16 -0400
>> 
EZ> You (or someone else) are welcome to take my ports and package them as
EZ> you see fit.  If you do a good job, I will even point to those
EZ> packages in nt/INSTALL etc.
>> 
>> Your work is very good and OK with me (although I can work with you to
>> automate the builds).  I am talking about distributing these binary
>> libraries better, especially critical updates.

EZ> That's exactly what I was talking about, too: take the port packaged
EZ> as drop-in zip files, and package it in some other way you consider to
EZ> be better.

That's great!  I'd love to make it a GNU ELPA package to be installed or
updated like any other such package.

EZ> As for critical updates, if you point me to the place or places where
EZ> these are described, I might consider whether it's time to update the
EZ> port.  (Please don't just point me to the gnutls-devel list, because I
EZ> read that, and have yet to see something like what you seem to allude
EZ> to.)

I would assume you want to automate the builds to some degree, but I
don't mind packaging whatever you produce.  Have you documented the
process anywhere so I can follow it if you can't do it?

There's several lists where critical vulnerabilities are disclosed but I
don't follow them all; currently this seems like the best way:
http://web.nvd.nist.gov/view/vuln/search-results?query=gnutls&search_type=all&cves=on&uscert_ta=on&uscert_vn=on&oval_query=on

Following gnutls-devel is probably sufficient for now and we can tell
the GnuTLS developers we are interested in early disclosure.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 25 May 2013 18:18:06 -0400
> 
> I would assume you want to automate the builds to some degree

When it comes to Windows, I don't believe in automated builds.

> Have you documented the process anywhere so I can follow it if you
> can't do it?

It's just the usual configure and make; the "interesting" parts begin
when one or both fail.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sun, 26 May 2013 05:52:52 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 25 May 2013 18:18:06 -0400
>> 
>> I would assume you want to automate the builds to some degree

EZ> When it comes to Windows, I don't believe in automated builds.

Thank you.  I won't pursue an automated W32 build further, just a
distribution mechanism for GnuTLS and possibly other libraries.

>> Have you documented the process anywhere so I can follow it if you
>> can't do it?

EZ> It's just the usual configure and make; the "interesting" parts begin
EZ> when one or both fail.

OK, got it.

Thanks
Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 25 May 2013 18:18:06 -0400
> 
> There's several lists where critical vulnerabilities are disclosed but I
> don't follow them all; currently this seems like the best way:
> http://web.nvd.nist.gov/view/vuln/search-results?query=gnutls&search_type=all&cves=on&uscert_ta=on&uscert_vn=on&oval_query=on

This seems to be very old.  This page looks much more relevant:

   http://gnutls.org/security.html

According to the latter, we are in good shape with 3.0.9.  I'll
probably look into updating to 3.2.x, once it stabilizes.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Sun, 26 May 2013 22:59:05 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 25 May 2013 18:18:06 -0400
>> 
>> There's several lists where critical vulnerabilities are disclosed but I
>> don't follow them all; currently this seems like the best way:
>> http://web.nvd.nist.gov/view/vuln/search-results?query=gnutls&search_type=all&cves=on&uscert_ta=on&uscert_vn=on&oval_query=on

EZ> This seems to be very old.  This page looks much more relevant:

EZ>    http://gnutls.org/security.html

There's also BUGTRAQ at http://www.securityfocus.com/archive/1

I don't know of a single place to track vulnerabilities, honestly.  They
seem to be all over the place.

EZ> According to the latter, we are in good shape with 3.0.9.  I'll
EZ> probably look into updating to 3.2.x, once it stabilizes.

OK.  That's good to know, and thank you for looking into it.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Stefan Monnier]

> those can be compromised too).  But we should at least make an effort to
> tell the user they are out of date, even if we do nothing more.
[...]
JB> I agree with Eli that installing GnuTLS from a zip file is easy enough.
> OK, 2 votes against and 1 in favor.

I haven't read this thread, so if you want my opinion, please explain
concisely what it's about (e.g. in which circumstance should we tell
the user (s)he's out of date).


        Stefan

GnuTLS updates proposal (was: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32)

[Ted Zlatanov]

On Fri, 24 May 2013 15:51:08 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> those can be compromised too).  But we should at least make an effort to
>> tell the user they are out of date, even if we do nothing more.
SM> [...]
JB> I agree with Eli that installing GnuTLS from a zip file is easy enough.
>> OK, 2 votes against and 1 in favor.

SM> I haven't read this thread, so if you want my opinion, please explain
SM> concisely what it's about (e.g. in which circumstance should we tell
SM> the user (s)he's out of date).

Generally library updates are announced on specialized lists that users
don't follow and a developer picks it up and pushes it out to the users.

I am proposing that GnuTLS updates should be pushed out this way (which
would also cover installation, so the user doesn't have to get it from a
ZIP file on W32 or from God knows where on Mac OS X...  Homebrew or Fink
or something else) instead of putting that responsibility on the user.

We could use the GNU ELPA as the transport (once it has package
signing).  I can be responsible for following the GnuTLS announcements.

This is not typical for Emacs so I've brought it up for discussion
instead of just doing the work.

So far Eli and Juanma are against my proposal.

Ted

Re: GnuTLS updates proposal

[Stefan Monnier]

> I am proposing that GnuTLS updates should be pushed out this way (which
> would also cover installation, so the user doesn't have to get it from a
> ZIP file on W32 or from God knows where on Mac OS X...  Homebrew or Fink
> or something else) instead of putting that responsibility on the user.

That sounds OK so far.

> We could use the GNU ELPA as the transport (once it has package
> signing).  I can be responsible for following the GnuTLS announcements.

Now I lost you.  How did we go from "gnutls" to "we"?
Shouldn't some gnutls guys take care of that?


        Stefan

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Fri, 24 May 2013 21:52:14 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I am proposing that GnuTLS updates should be pushed out this way (which
>> would also cover installation, so the user doesn't have to get it from a
>> ZIP file on W32 or from God knows where on Mac OS X...  Homebrew or Fink
>> or something else) instead of putting that responsibility on the user.

SM> That sounds OK so far.

Wonderful.

>> We could use the GNU ELPA as the transport (once it has package
>> signing).  I can be responsible for following the GnuTLS announcements.

SM> Now I lost you.  How did we go from "gnutls" to "we"?
SM> Shouldn't some gnutls guys take care of that?

So far they have not and it's not a priority to them.  Eli did a lot of
work cleaning up the code and getting it to compile--it's thanks to him
that Emacs has a recent GnuTLS DLL.

If you want, I can ask again and see what they say now... I'm not
optimistic.

So, as far as building the GnuTLS DLLs, I'm hopeful we'll find a way to
automate that but not hopeful that it will happen on the GnuTLS
maintainers' side.  Building them on Mac OS X is trivial.  This is the
"hard work" part and Eli already pointed out we'll need volunteers and
resources.

Whoever provides the actual DLLs, there has to be a package maintainer
for the GNU ELPA package of GnuTLS; I will do it unless there's someone
else excited by the prospect of tracking security announcements.  Plus,
of course, we can't rely on the GNU ELPA for library updates until the
package signing work is done.

Ted

Re: GnuTLS updates proposal

[Stefan Monnier]

SM> Now I lost you.  How did we go from "gnutls" to "we"?
SM> Shouldn't some gnutls guys take care of that?

> So far they have not and it's not a priority to them.  Eli did a lot of
> work cleaning up the code and getting it to compile--it's thanks to him
> that Emacs has a recent GnuTLS DLL.

So, why should this fall on us?  I mean, is Emacs the only program
using gnutls?
If so, why?  what do other programs use?
If not, what do the other programs do about it?

> So, as far as building the GnuTLS DLLs, I'm hopeful we'll find a way to
> automate that but not hopeful that it will happen on the GnuTLS
> maintainers' side.  Building them on Mac OS X is trivial.  This is the
> "hard work" part and Eli already pointed out we'll need volunteers and
> resources.

Ah, so the problem is not only for Windows but also for Mac OS X?

> Whoever provides the actual DLLs, there has to be a package maintainer
> for the GNU ELPA package of GnuTLS;

As a user, I wouldn't like it for a program such as Emacs to decide it
will take over responsibility for a generic library used by other
programs, so I find the idea of distributing it via GNU ELPA rather odd.


        Stefan

Re: GnuTLS updates proposal

[Stephen J. Turnbull]

Stefan Monnier writes:

 > SM> Now I lost you.  How did we go from "gnutls" to "we"?
 > SM> Shouldn't some gnutls guys take care of that?

No, why should they?  Emacs doesn't provide Debian or Red Hat
packages, the downstream does that.  When Emacs is downstream, it
will be expected to do the work, if anybody does.

 > > So far they have not and it's not a priority to them.  Eli did a lot of
 > > work cleaning up the code and getting it to compile--it's thanks to him
 > > that Emacs has a recent GnuTLS DLL.
 > 
 > So, why should this fall on us?  I mean, is Emacs the only program
 > using gnutls?

Wrong question.  The question is does Emacs (which is a collective
work drawing on many packages not developed here) use GnuTLS?  If so,
is there an existing distribution of GnuTLS that "enough" users use so
that it's preferable to ask the users who don't have it to download it
themselves, rather than include it in the Emacs (or ELPA) distribution?

The debate here is about the value of "enough".  Ted wants much closer
to 100% than Eli and Juanma think necessary.

 > If so, why?  what do other programs use?

Most programs I know of use OpenSSL.  There are political problems,
and maybe license problems (the EAY clause) with Emacs doing that (but
see the list of OpenSSL users below).  From my MacPorts installation:

MacPorts 14:40$ port dependents gnutls
gnutls has no dependents.

(My build of the Emacs trunk depends on gnutls, but not the MacPorts
version I currently have installed.)

MacPorts 15:03$ port dependents openssl
cmake depends on openssl
curl depends on openssl
cyrus-sasl2 depends on openssl
git-core depends on openssl
gnome-vfs depends on openssl
kerberos5 depends on openssl
neon depends on openssl
openldap depends on openssl
openssh depends on openssl
python26 depends on openssl
python27 depends on openssl
python32 depends on openssl
python33 depends on openssl
qt4-mac depends on openssl
serf0 depends on openssl
serf1 depends on openssl

Note that these are only the installed packages (so it's biased to my
taste), and there are several cases of multiple versions (but the
ratio is nevertheless damning).  Note that even gnome-vfs depends on
OpenSSL.

 > If not, what do the other programs do about it?

Avoid it, apparently.

 > Ah, so the problem is not only for Windows but also for Mac OS X?

The problem exists for all OSes.  Eg, Debian stable is probably more
out of date than either of the above (until sometime in the next few
days if I'm reading the tea leaves right).

However, aggregating Mac and Windows here should be done with great
care (see below).

 > As a user, I wouldn't like it for a program such as Emacs to decide it
 > will take over responsibility for a generic library used by other
 > programs, so I find the idea of distributing it via GNU ELPA rather odd.

That's because you don't use Windows.  Windows users are used to that,
and in fact prefer it (if done right), because many programs have
problems with the "wrong" version of certain libraries, so they bundle
them.

Mac, however, is much closer to the *nix ideal of one instance per
package, so I suspect a lot of Mac users would prefer to use their
system version of gnutls, whereever it comes from (mine comes from
MacPorts).  But "DLL Hell" is a pain on many *nix systems as well (eg,
my Gentoo update is currently blocked because SBCL demands one version
of certain libraries, while Gentoo wants to install more recent ones).
It is also common in Python applications (MacPorts installs a separate
compilation of each library I install for all interpreter versions I
have -- I need 3 to support various applications, plus the most recent
release for development).

Heck, Emacs (until the advent of ELPA) was perhaps the premier example
of this behavior: distribute all the Lisp anyone could ever want, so
that the developers could have a free hand in refactoring.  XEmacs
proposed (on several occasions) that one of the existing systems
(XEmacs and the OSU repository are two I remember, the latter not
really being a package system but rather a classification system) but
were refused.  The reason given was that all Lisp was part of Emacs,
and Emacs could only afford to support one version of the language at
a time so a package system made no sense: it would constrain the
language development too much.

That said, indeed there are Mac users who expect full-blown
installers.  But I would expect them, by the same token, to mostly be
using Aquamacs.  In that case you can leave it up to the Aquamacs
maintainer(s), and the issue is restricted to Windows.

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Sat, 25 May 2013 00:47:41 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> Now I lost you.  How did we go from "gnutls" to "we"?
SM> Shouldn't some gnutls guys take care of that?

>> So far they have not and it's not a priority to them.  Eli did a lot of
>> work cleaning up the code and getting it to compile--it's thanks to him
>> that Emacs has a recent GnuTLS DLL.

SM> So, why should this fall on us?  I mean, is Emacs the only program
SM> using gnutls?
SM> If so, why?  what do other programs use?
SM> If not, what do the other programs do about it?

GnuTLS is not a popular solution.  OpenSSL is far, far more popular and
well supported on W32 and other platforms.  So there isn't a good sample
size to answer that question.

In addition, Emacs itself is not an application, it's more of a platform
IMO.  That makes it a little more responsible for keeping its libraries
up to date than most applications, again IMHO.

>> So, as far as building the GnuTLS DLLs, I'm hopeful we'll find a way to
>> automate that but not hopeful that it will happen on the GnuTLS
>> maintainers' side.  Building them on Mac OS X is trivial.  This is the
>> "hard work" part and Eli already pointed out we'll need volunteers and
>> resources.

SM> Ah, so the problem is not only for Windows but also for Mac OS X?

Yes, because on Mac OS X there is no authoritative package manager
either.

>> Whoever provides the actual DLLs, there has to be a package maintainer
>> for the GNU ELPA package of GnuTLS;

SM> As a user, I wouldn't like it for a program such as Emacs to decide it
SM> will take over responsibility for a generic library used by other
SM> programs, so I find the idea of distributing it via GNU ELPA rather odd.

If it was a program, sure.  The closest analogue to Emacs on W32 is
Cygwin as a whole, which does provide its own updates.

I realize my proposal is not the way we've always done things and seems
odd and unusual.  My goal right now is to collect votes for or against
my proposal, especially from the maintainers.  If everyone but me says
it's a bad idea, then it probably is.

Ted

Re: GnuTLS updates proposal

[Richard Stallman]

Emacs cannot link with OpenSSL due to license incompatibility.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Sun, 26 May 2013 00:51:01 -0400 Richard Stallman <rms@gnu.org> wrote: 

RS> Emacs cannot link with OpenSSL due to license incompatibility.

Of course.  Stefan was asking how other applications deal with the same
problem, and I mentioned OpenSSL because it's the closest analogue to
GnuTLS.

Ted

Re: GnuTLS updates proposal (was: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32)

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 24 May 2013 17:53:36 -0400
> 
> So far Eli and Juanma are against my proposal.

Correction: they think that the current arrangement, whereby users are
pointed to specific distributions of GnuTLS and other 3rd party
packages, are adequate enough, and any additional effort is something
the Emacs developers are unlikely to be able to sustain.

Re: GnuTLS updates proposal

[Stefan Monnier]

> Correction: they think that the current arrangement, whereby users are
> pointed to specific distributions of GnuTLS and other 3rd party
> packages, are adequate enough, and any additional effort is something
> the Emacs developers are unlikely to be able to sustain.

I sure won't spend any time on this, indeed, and I hope you don't
either.  But I guess if someone wants to provide some Elisp code that
eases up the installation (taking care of downloading the right DLL
(which I guess might depend on the Emacs version and even the specific
build) and installing it at the right place), that would be OK and could
very well fit in GNU ELPA (I don't think I'd want the DLL itself to be
there).  Hopefully such code would handle not just gnutls but also the
other libraries, like libxml.


        Stefan

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Sat, 25 May 2013 03:10:36 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Correction: they think that the current arrangement, whereby users are
>> pointed to specific distributions of GnuTLS and other 3rd party
>> packages, are adequate enough, and any additional effort is something
>> the Emacs developers are unlikely to be able to sustain.

SM> I sure won't spend any time on this, indeed, and I hope you don't
SM> either.  But I guess if someone wants to provide some Elisp code that
SM> eases up the installation (taking care of downloading the right DLL
SM> (which I guess might depend on the Emacs version and even the specific
SM> build) and installing it at the right place), that would be OK and could
SM> very well fit in GNU ELPA (I don't think I'd want the DLL itself to be
SM> there).  Hopefully such code would handle not just gnutls but also the
SM> other libraries, like libxml.

Sounds good, I'll put it on my TODO list after ELPA package signing.
Thank you for looking at my proposal.

Ted

Re: GnuTLS updates proposal

[Ted Zlatanov]

On Sun, 26 May 2013 17:34:10 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Sat, 25 May 2013 03:10:36 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>>> Correction: they think that the current arrangement, whereby users are
>>> pointed to specific distributions of GnuTLS and other 3rd party
>>> packages, are adequate enough, and any additional effort is something
>>> the Emacs developers are unlikely to be able to sustain.

SM> I sure won't spend any time on this, indeed, and I hope you don't
SM> either.  But I guess if someone wants to provide some Elisp code that
SM> eases up the installation (taking care of downloading the right DLL
SM> (which I guess might depend on the Emacs version and even the specific
SM> build) and installing it at the right place), that would be OK and could
SM> very well fit in GNU ELPA (I don't think I'd want the DLL itself to be
SM> there).  Hopefully such code would handle not just gnutls but also the
SM> other libraries, like libxml.

TZ> Sounds good, I'll put it on my TODO list after ELPA package signing.
TZ> Thank you for looking at my proposal.

Update: I talked to the GnuTLS developers.  They now have W32 binaries
but do *not* guarantee anything about them except their internal tests
have passed, and don't consider W32 an important platform.  So the Emacs
W32 integration could fail and they will still release those DLLs.  But
at least the W32 build is considered somewhat important by them.

It seems that it's better to keep relying on Eli and myself as the
build+QA team for the Emacs side, and to keep using Eli's binaries, with
me as the fallback if he can't produce them.  So basically this does not
change the plans we discussed earlier, but we now have more assurance of
a working W32 baseline from the GnuTLS developers.

Ted

Re: GnuTLS updates proposal (was: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32)

[Juanma Barranquero]

On Sat, May 25, 2013 at 8:39 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> Correction: they think that the current arrangement, whereby users are
> pointed to specific distributions of GnuTLS and other 3rd party
> packages, are adequate enough, and any additional effort is something
> the Emacs developers are unlikely to be able to sustain.

That's exactly my opinion, yeah.

   J

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Mon, 20 May 2013 05:58:40 -0400
> 
> On Mon, 20 May 2013 04:08:57 +0200 Juanma Barranquero <lekktu@gmail.com> wrote: 
> 
> JB> On Mon, May 20, 2013 at 1:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> >> It would mean changing the way Mac OS X and W32 Emacs
> >> builds are distributed, to include the GnuTLS libraries with the build,
> >> and we'd have to implement a way (perhaps through the ELPA) to
> >> distribute updates to these libraries.
> 
> JB> Why would that be a good idea now if it wasn't seen as such before?
> 
> Because we've had years of experience supporting GnuTLS, we're moving to
> GnuTLS 3.x soon

What do you mean by "moving to GnuTLS 3.x soon"?  For Windows users,
the recommended GnuTLS port is of v3.0.9; isn't that "3.x" already?

> ELPA has matured and requires a secure transport[1], and we've heard
> many times from our users on those two platforms that we should.
> 
> The current situation is worse: users are on their own and the GnuTLS
> developers have not given them a way to auto-update or indicated
> interest in doing so (I asked last year IIRC, and see
> http://josefsson.org/gnutls4win/)
> 
> Ted
> 
> [1] I also proposed the ELPA as the way to distribute GnuTLS updates.
> There's a separate thread about signing ELPA packages which would let us
> avoid using a broken GnuTLS to pull its own updates, but that's not
> essential to my proposal.

Sorry, I'm confused: are you talking about securing Emacs or about
securing ELPA?  You mix these two (which are quite different issues
with very different implications) in ways that make it hard to
understand what are you saying.

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 19:17:45 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Mon, 20 May 2013 05:58:40 -0400
>> 
>> On Mon, 20 May 2013 04:08:57 +0200 Juanma Barranquero <lekktu@gmail.com> wrote: 
>> 
JB> On Mon, May 20, 2013 at 1:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> >> It would mean changing the way Mac OS X and W32 Emacs
>> >> builds are distributed, to include the GnuTLS libraries with the build,
>> >> and we'd have to implement a way (perhaps through the ELPA) to
>> >> distribute updates to these libraries.
>> 
JB> Why would that be a good idea now if it wasn't seen as such before?
>> 
>> Because we've had years of experience supporting GnuTLS, we're moving to
>> GnuTLS 3.x soon

EZ> What do you mean by "moving to GnuTLS 3.x soon"?  For Windows users,
EZ> the recommended GnuTLS port is of v3.0.9; isn't that "3.x" already?

We use the 2.x API which works with 3.x.

>> [1] I also proposed the ELPA as the way to distribute GnuTLS updates.
>> There's a separate thread about signing ELPA packages which would let us
>> avoid using a broken GnuTLS to pull its own updates, but that's not
>> essential to my proposal.

EZ> Sorry, I'm confused: are you talking about securing Emacs or about
EZ> securing ELPA?  You mix these two (which are quite different issues
EZ> with very different implications) in ways that make it hard to
EZ> understand what are you saying.

There is nothing about securing Emacs itself here (that topic includes
secure credential storage and such, which are not relevant here).

I'm talking about two things:

1) distributing GnuTLS with Emacs and also distributing updates to it.

2) possibly using the ELPA to distribute the GnuTLS updates as a package
that's signed.

Ted

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

On Mon, May 20, 2013 at 12:05 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>
> Using an external binary to transport SSL or TLS is a hack IMO.

I see, and reading a bit of tls.el it makes sense. It does all this
crazy regexp searching... This is a hack in my book too, albeit one
that's been working fine,

> Did you propose a patch?  I would commit a patch but can't write it
> despite your great description of the problem.
>

Here is a patch that should be equivalent to the defadvice I'm using.
As I said, it works for me. Also I didn't have a VC copy of emacs so I
used  `diff-buffer-with-file'

diff -u -L /usr/local/share/emacs/24.3/lisp/net/tls.el.gz -L
\#\<buffer\ tls.el.gz\> /tmp/jka-com1909LVh
/tmp/buffer-content-1909lpt
--- /usr/local/share/emacs/24.3/lisp/net/tls.el.gz
+++ #<buffer tls.el.gz>
@@ -286,7 +286,11 @@
 			     (format "Host name in certificate doesn't \
 match `%s'. Connect anyway? " host))))))
 	(setq done nil)
-	(delete-process process)))
+	(delete-process process))
+      ;; delete all the informational messages that could
+      ;; confuse futures users of `buffer'
+      ;;
+      (delete-region (point-min) (point)))
     (message "Opening TLS connection to `%s'...%s"
 	     host (if done "done" "failed"))
     (when use-temp-buffer


João
-- 
João Távora

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[Ted Zlatanov]

On Mon, 20 May 2013 23:07:30 +0100 João Távora <joaotavora@gmail.com> wrote: 

JT> Here is a patch that should be equivalent to the defadvice I'm using.
JT> As I said, it works for me. Also I didn't have a VC copy of emacs so I
JT> used  `diff-buffer-with-file'

I applied something very similar.  It's a tiny change so it doesn't
require assignment papers.  If you'd like to contribute to GNU Emacs or
other GNU software in the future, I encourage you to file them.

revno: 112858
timestamp: Wed 2013-06-05 11:03:37 -0400
message:
  (open-tls-stream): Remove unneeded buffer contents when opening the connection.

This change didn't break anything in my testing.  Please test so I can
close the bug if it's fixed for you.

Ted

diff -u -L /usr/local/share/emacs/24.3/lisp/net/tls.el.gz -L
\#\<buffer\ tls.el.gz\> /tmp/jka-com1909LVh
/tmp/buffer-content-1909lpt
--- /usr/local/share/emacs/24.3/lisp/net/tls.el.gz
+++ #<buffer tls.el.gz>
@@ -286,7 +286,11 @@
			     (format "Host name in certificate doesn't \
 match `%s'. Connect anyway? " host))))))
	(setq done nil)
-	(delete-process process)))
+	(delete-process process))
+      ;; delete all the informational messages that could
+      ;; confuse futures users of `buffer'
+      ;;
+      (delete-region (point-min) (point)))
     (message "Opening TLS connection to `%s'...%s"
	     host (if done "done" "failed"))
     (when use-temp-buffer

Re: bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

Thanks!

To test, I will load just load the trunks "tls.el" in my existing W32
emacs. (I don't have time to get a windows emacs-building toolchain).

As Stefan mentioned, I have already signed papers.

João

On Wed, Jun 5, 2013 at 4:06 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Mon, 20 May 2013 23:07:30 +0100 João Távora <joaotavora@gmail.com> wrote:
>
> JT> Here is a patch that should be equivalent to the defadvice I'm using.
> JT> As I said, it works for me. Also I didn't have a VC copy of emacs so I
> JT> used  `diff-buffer-with-file'
>
> I applied something very similar.  It's a tiny change so it doesn't
> require assignment papers.  If you'd like to contribute to GNU Emacs or
> other GNU software in the future, I encourage you to file them.
>
> revno: 112858
> timestamp: Wed 2013-06-05 11:03:37 -0400
> message:
>   (open-tls-stream): Remove unneeded buffer contents when opening the connection.
>
> This change didn't break anything in my testing.  Please test so I can
> close the bug if it's fixed for you.
>
> Ted
>
> diff -u -L /usr/local/share/emacs/24.3/lisp/net/tls.el.gz -L
> \#\<buffer\ tls.el.gz\> /tmp/jka-com1909LVh
> /tmp/buffer-content-1909lpt
> --- /usr/local/share/emacs/24.3/lisp/net/tls.el.gz
> +++ #<buffer tls.el.gz>
> @@ -286,7 +286,11 @@
>                              (format "Host name in certificate doesn't \
>  match `%s'. Connect anyway? " host))))))
>         (setq done nil)
> -       (delete-process process)))
> +       (delete-process process))
> +      ;; delete all the informational messages that could
> +      ;; confuse futures users of `buffer'
> +      ;;
> +      (delete-region (point-min) (point)))
>      (message "Opening TLS connection to `%s'...%s"
>              host (if done "done" "failed"))
>      (when use-temp-buffer
>



-- 
João Távora