Win32 GnuTLS DLL installer?

Win32 GnuTLS DLL installer?

[Ted Zlatanov]

It would be nice to have a Win32 installer for Emacs that downloaded the
GnuTLS DLLs.

Or maybe they can be packaged with the Win32 binaries. I don't know the
best solution.

Ted

Re: Win32 GnuTLS DLL installer?

[Phillip Lord]

On Wed, September 20, 2017 4:08 pm, Ted Zlatanov wrote:
> It would be nice to have a Win32 installer for Emacs that downloaded the
> GnuTLS DLLs.
>
>
> Or maybe they can be packaged with the Win32 binaries. I don't know the
> best solution.


They are in https://ftp.gnu.org/gnu/emacs/windows/emacs-25-x86_64-deps.zip.

For emacs-26, I hope to start releasing two sets of zip files -- "emacs"
and "emacs-with-deps". The later will mean that you won't need to take two
packages, just unzip and go.

Actually, this will mean 4 packages -- 32 and 64 bit. Oh dear.

Phil

Re: Win32 GnuTLS DLL installer?

[Sivaram Neelakantan]

On Wed, Sep 20 2017,Phillip Lord wrote:


[snipped 9 lines]

> They are in https://ftp.gnu.org/gnu/emacs/windows/emacs-25-x86_64-deps.zip.
>
> For emacs-26, I hope to start releasing two sets of zip files -- "emacs"
> and "emacs-with-deps". The later will mean that you won't need to take two
> packages, just unzip and go.

As a user "emacs-with-deps" single zip file would be a tremendous
help.  I spend a fair amount of time trying to get Emacs and cygwin to
play along, then give up, then harass Eli for help and then try some
random stuff from the Web and then get continually surprised by
strange behaviour of cygwin paths and win Emacs/bin file clashes.

Make it simple so that users have only themselves to blame. :-)

Thanks

sivaram
-- 

Re: Win32 GnuTLS DLL installer?

[Eli Zaretskii]

> Date: Wed, 20 Sep 2017 20:15:33 -0000
> From: "Phillip Lord" <phillip.lord@russet.org.uk>
> 
> > Or maybe they can be packaged with the Win32 binaries. I don't know the
> > best solution.
> 
> They are in https://ftp.gnu.org/gnu/emacs/windows/emacs-25-x86_64-deps.zip.
> 
> For emacs-26, I hope to start releasing two sets of zip files -- "emacs"
> and "emacs-with-deps".

When you do, please update nt/README.W32 (and the README file you
upload to the GNU FTP site) with the information about the new
structure of the binary zip files.

Thanks.

Re: Win32 GnuTLS DLL installer?

[Ted Zlatanov]

On Wed, 20 Sep 2017 20:15:33 -0000 "Phillip Lord" <phillip.lord@russet.org.uk> wrote: 

PL> On Wed, September 20, 2017 4:08 pm, Ted Zlatanov wrote:
>> It would be nice to have a Win32 installer for Emacs that downloaded the
>> GnuTLS DLLs.
>> 
>> 
>> Or maybe they can be packaged with the Win32 binaries. I don't know the
>> best solution.

PL> They are in https://ftp.gnu.org/gnu/emacs/windows/emacs-25-x86_64-deps.zip.

PL> For emacs-26, I hope to start releasing two sets of zip files -- "emacs"
PL> and "emacs-with-deps". The later will mean that you won't need to take two
PL> packages, just unzip and go.

PL> Actually, this will mean 4 packages -- 32 and 64 bit. Oh dear.

Please do. Thank you.

(A possible use case for the security-patches package/repo/mechanism
could be to check the DLL version and notify the user if it's outdated.
I don't know if this is possible or built-in in W32 but it seems to be
the responsibility of the software that installs the DLL.)

Thanks
Ted

Re: Win32 GnuTLS DLL installer?

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 21 Sep 2017 10:28:03 -0400
> 
> (A possible use case for the security-patches package/repo/mechanism
> could be to check the DLL version and notify the user if it's outdated.

How do you define "outdated"?  GnuTLS developers maintain 3 branches
in parallel, and release versions from all the 3 branches.  Which one
should we follow, and does any N+1 release from that branch mean that
the N release is considered "outdated" and should be replaced?

Moreover, accommodating a new version of GnuTLS might mean changes to
Emacs C and/or Lisp sources -- are we going to release patches to
the core sources through ELPA or something?

IOW, this sounds like a major undertaking on our part, and I'm sure
more and more issues will pop up as we consider the implications.  I'm
not sure we want to become a de-facto "distro" for MS-Windows users,
as I don't think we have the resources, even if we have the desire.

Re: Win32 GnuTLS DLL installer?

[Ted Zlatanov]

On Thu, 21 Sep 2017 17:49:32 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Thu, 21 Sep 2017 10:28:03 -0400
>> 
>> (A possible use case for the security-patches package/repo/mechanism
>> could be to check the DLL version and notify the user if it's outdated.

EZ> How do you define "outdated"?  GnuTLS developers maintain 3 branches
EZ> in parallel, and release versions from all the 3 branches.  Which one
EZ> should we follow, and does any N+1 release from that branch mean that
EZ> the N release is considered "outdated" and should be replaced?

That's usually the platform's responsibility, but in this case it seems
up to us. I would keep up with the current branch (whatever was
installed already) at least, on the principle of least surprise.

EZ> Moreover, accommodating a new version of GnuTLS might mean changes to
EZ> Emacs C and/or Lisp sources -- are we going to release patches to
EZ> the core sources through ELPA or something?

Theoretically the current branch will not require that. I would just
post a non-intrusive message to the user for now.

EZ> IOW, this sounds like a major undertaking on our part, and I'm sure
EZ> more and more issues will pop up as we consider the implications.  I'm
EZ> not sure we want to become a de-facto "distro" for MS-Windows users,
EZ> as I don't think we have the resources, even if we have the desire.

Agreed, I wasn't suggesting all of that. It does seem like a general
Emacs update process may be more appropriate, like Cygwin does, but
simple messaging is a lot easier and less risky.

Ted

Re: Win32 GnuTLS DLL installer?

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 21 Sep 2017 11:38:01 -0400
> 
> EZ> How do you define "outdated"?  GnuTLS developers maintain 3 branches
> EZ> in parallel, and release versions from all the 3 branches.  Which one
> EZ> should we follow, and does any N+1 release from that branch mean that
> EZ> the N release is considered "outdated" and should be replaced?
> 
> That's usually the platform's responsibility, but in this case it seems
> up to us. I would keep up with the current branch (whatever was
> installed already) at least, on the principle of least surprise.

So whenever the current branch sees a new release, someone would have
to port it to Windows and provide the binaries?

> EZ> Moreover, accommodating a new version of GnuTLS might mean changes to
> EZ> Emacs C and/or Lisp sources -- are we going to release patches to
> EZ> the core sources through ELPA or something?
> 
> Theoretically the current branch will not require that.

Theoretically, yes.  In practice, this can and did happen.

> I would just post a non-intrusive message to the user for now.

Not sure how a message could help with source-level changes.  People
who track the Git repository normally don't keep local patches.
People who use official releases don't even have the sources in most
cases.

> EZ> IOW, this sounds like a major undertaking on our part, and I'm sure
> EZ> more and more issues will pop up as we consider the implications.  I'm
> EZ> not sure we want to become a de-facto "distro" for MS-Windows users,
> EZ> as I don't think we have the resources, even if we have the desire.
> 
> Agreed, I wasn't suggesting all of that. It does seem like a general
> Emacs update process may be more appropriate, like Cygwin does, but
> simple messaging is a lot easier and less risky.

Again, I'm not sure I see how a message would help.  And Cygwin does
produce packages as GNU/Linux distribution do, something I don't think
we should take upon ourselves.

Re: Win32 GnuTLS DLL installer?

[Ted Zlatanov]

On Thu, 21 Sep 2017 19:58:21 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Thu, 21 Sep 2017 11:38:01 -0400
>> 
EZ> How do you define "outdated"?  GnuTLS developers maintain 3 branches
EZ> in parallel, and release versions from all the 3 branches.  Which one
EZ> should we follow, and does any N+1 release from that branch mean that
EZ> the N release is considered "outdated" and should be replaced?
>> 
>> That's usually the platform's responsibility, but in this case it seems
>> up to us. I would keep up with the current branch (whatever was
>> installed already) at least, on the principle of least surprise.

EZ> So whenever the current branch sees a new release, someone would have
EZ> to port it to Windows and provide the binaries?

I'd check for the binaries that are actually available, so it wouldn't
do anything other than saying "hey there's a new DLL you should run."
And there would be no new process on the build side.

EZ> Moreover, accommodating a new version of GnuTLS might mean changes to
EZ> Emacs C and/or Lisp sources -- are we going to release patches to
EZ> the core sources through ELPA or something?
>> 
>> Theoretically the current branch will not require that.

EZ> Theoretically, yes.  In practice, this can and did happen.

You're right. I don't see a way around that except tracking version
compatibility in a security-patches package or repo OR accepting that
rare problem as a known and acceptable situation.

>> I would just post a non-intrusive message to the user for now.

EZ> Not sure how a message could help with source-level changes.  People
EZ> who track the Git repository normally don't keep local patches.
EZ> People who use official releases don't even have the sources in most
EZ> cases.

Right. I'd just check the DLL binaries actually available.

A message would help the user determine that they should upgrade.

Ted

Re: Win32 GnuTLS DLL installer?

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 21 Sep 2017 13:33:40 -0400
> 
> EZ> So whenever the current branch sees a new release, someone would have
> EZ> to port it to Windows and provide the binaries?
> 
> I'd check for the binaries that are actually available, so it wouldn't
> do anything other than saying "hey there's a new DLL you should run."

Available from where?  Surely, we won't want to recommend
security-related DLLs whose quality we cannot guarantee, would we?
And how can we guarantee quality of DLLs built by someone out there,
about whose build procedure and its QA we know nothing?  E.g., how do
we know they've run the test suite and made sure every failure is
either fixed or analyzed to conclude that it isn't a real problem?

Bottom line, I'd really love to see someone volunteer to do this job
in a way that we could simply rely on them and point to their site (or
copy stuff from there to ELPA), but I'm not holding my breath, having
done that several times myself.  It's not an easy job, and requires
non-trivial investment of time and effort.

Re: Win32 GnuTLS DLL installer?

[Ted Zlatanov]

On Thu, 21 Sep 2017 20:44:05 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Thu, 21 Sep 2017 13:33:40 -0400
>> 
EZ> So whenever the current branch sees a new release, someone would have
EZ> to port it to Windows and provide the binaries?
>> 
>> I'd check for the binaries that are actually available, so it wouldn't
>> do anything other than saying "hey there's a new DLL you should run."

EZ> Available from where?  Surely, we won't want to recommend
EZ> security-related DLLs whose quality we cannot guarantee, would we?

As Phillip said,
https://ftp.gnu.org/gnu/emacs/windows/emacs-25-x86_64-deps.zip and
whatever other locations on that server are appropriate. I think that's
the simplest, least surprising solution.

If we track a single ZIP file, then we may have to look inside for the
DLL version specifically. But maybe it's poassible to agree on
https://ftp.gnu.org/gnu/emacs/windows/some-special-release-file.txt and
ask whoever builds the W32 binaries update it.

EZ> Bottom line, I'd really love to see someone volunteer to do this job
EZ> in a way that we could simply rely on them and point to their site (or
EZ> copy stuff from there to ELPA), but I'm not holding my breath, having
EZ> done that several times myself.  It's not an easy job, and requires
EZ> non-trivial investment of time and effort.

I understand your concerns. They apply equally to the rest of the W32
binaries and dependencies and I'm not ignoring them.

When a GitLab server is available, maybe we can set up a W32 build slave
to build and test these binaries.

Ted

Re: Win32 GnuTLS DLL installer?

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 21 Sep 2017 13:57:46 -0400
> 
> EZ> Available from where?  Surely, we won't want to recommend
> EZ> security-related DLLs whose quality we cannot guarantee, would we?
> 
> As Phillip said,
> https://ftp.gnu.org/gnu/emacs/windows/emacs-25-x86_64-deps.zip and
> whatever other locations on that server are appropriate. I think that's
> the simplest, least surprising solution.

It may be the simplest, but are they good enough for a dedicated,
security-related package?  I'm not sure.  E.g., do the MSYS2 people,
who produce the DLLs which Phillip repackages, habitually run the test
suite of those DLLs, and investigate every failure?

> EZ> Bottom line, I'd really love to see someone volunteer to do this job
> EZ> in a way that we could simply rely on them and point to their site (or
> EZ> copy stuff from there to ELPA), but I'm not holding my breath, having
> EZ> done that several times myself.  It's not an easy job, and requires
> EZ> non-trivial investment of time and effort.
> 
> I understand your concerns. They apply equally to the rest of the W32
> binaries and dependencies and I'm not ignoring them.

They do, but producing a security-catering package brings on
additional concerns.  And I'm not sure our simple practices are up to
the challenge.

> When a GitLab server is available, maybe we can set up a W32 build slave
> to build and test these binaries.

That'd be good progress, but someone will still have to review the
failures in the optional libraries and fix them.  The upstream
developers only do that for GNU/Linux builds.

Re: Win32 GnuTLS DLL installer?

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 20 Sep 2017 20:15:33 -0000
>> From: "Phillip Lord" <phillip.lord@russet.org.uk>
>> 
>> > Or maybe they can be packaged with the Win32 binaries. I don't know the
>> > best solution.
>> 
>> They are in https://ftp.gnu.org/gnu/emacs/windows/emacs-25-x86_64-deps.zip.
>> 
>> For emacs-26, I hope to start releasing two sets of zip files -- "emacs"
>> and "emacs-with-deps".
>
> When you do, please update nt/README.W32 (and the README file you
> upload to the GNU FTP site) with the information about the new
> structure of the binary zip files.


Yes, I will.

Phil

Re: Win32 GnuTLS DLL installer?

[Phillip Lord]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 21 Sep 2017 17:49:32 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
>
> EZ> IOW, this sounds like a major undertaking on our part, and I'm sure
> EZ> more and more issues will pop up as we consider the implications.  I'm
> EZ> not sure we want to become a de-facto "distro" for MS-Windows users,
> EZ> as I don't think we have the resources, even if we have the desire.
>
> Agreed, I wasn't suggesting all of that. It does seem like a general
> Emacs update process may be more appropriate, like Cygwin does, but
> simple messaging is a lot easier and less risky.


At the moment, we struggle to produce a nicely packaged windows binary,
on time after every release (by "we" I mean "I" of course). Probably, I
should talk to Chris Zheng (http://zklhp.github.io/emacs-w64/), he seems
a bit quicker than me.

To me, the sane "update" proceedure is "install a new version of
emacs". If you want updating outside of that there are more holisitic
solutions like chocolaty.

Phil

Re: Win32 GnuTLS DLL installer?

[Phillip Lord]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 21 Sep 2017 20:44:05 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
> EZ> Bottom line, I'd really love to see someone volunteer to do this job
> EZ> in a way that we could simply rely on them and point to their site (or
> EZ> copy stuff from there to ELPA), but I'm not holding my breath, having
> EZ> done that several times myself.  It's not an easy job, and requires
> EZ> non-trivial investment of time and effort.
>
> I understand your concerns. They apply equally to the rest of the W32
> binaries and dependencies and I'm not ignoring them.
>
> When a GitLab server is available, maybe we can set up a W32 build slave
> to build and test these binaries.


That would be a joy. At the moment, the binaries have minimal testing.

We don't have a good test suite, though, not for binaries. For example,
if I don't add libXpm into the zip file, the tests will probably
succeed, Emacs will run, but it will look very ropey.

Phil

Re: Win32 GnuTLS DLL installer?

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Thu, 21 Sep 2017 13:57:46 -0400
>> 
>> EZ> Available from where?  Surely, we won't want to recommend
>> EZ> security-related DLLs whose quality we cannot guarantee, would we?
>> 
>> As Phillip said,
>> https://ftp.gnu.org/gnu/emacs/windows/emacs-25-x86_64-deps.zip and
>> whatever other locations on that server are appropriate. I think that's
>> the simplest, least surprising solution.
>
> It may be the simplest, but are they good enough for a dedicated,
> security-related package?  I'm not sure.  E.g., do the MSYS2 people,
> who produce the DLLs which Phillip repackages, habitually run the test
> suite of those DLLs, and investigate every failure?

Given that these are the DLLs than any user of Emacs is likely to update
their libgnutls from anyway, then this doesn't strike me as a major
problem. I mean, it's as good a solution as any.

Phil

>> When a GitLab server is available, maybe we can set up a W32 build slave
>> to build and test these binaries.
>
> That'd be good progress, but someone will still have to review the
> failures in the optional libraries and fix them.  The upstream
> developers only do that for GNU/Linux builds.

Re: Win32 GnuTLS DLL installer?

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Ted Zlatanov <tzz@lifelogs.com>,  emacs-devel@gnu.org
> Date: Thu, 21 Sep 2017 22:16:36 +0100
> 
> > It may be the simplest, but are they good enough for a dedicated,
> > security-related package?  I'm not sure.  E.g., do the MSYS2 people,
> > who produce the DLLs which Phillip repackages, habitually run the test
> > suite of those DLLs, and investigate every failure?
> 
> Given that these are the DLLs than any user of Emacs is likely to update
> their libgnutls from anyway, then this doesn't strike me as a major
> problem. I mean, it's as good a solution as any.

I'm quite ignorant about security issues, but one thing I did learn is
that "as good solution as any" is usually not good enough for
security-critical issues.

I can tell that all the DLL binaries on the ezwinports site always
pass their test suite, and all the test failures are looked into and
fixed if the failures are real.  That is why I generally update GnuTLS
and other libraries relatively infrequently: it's a serious job that
takes time and energy.

If we are willing to disregard the QA in these libraries for the
Windows users, then I submit that we should not consider seriously any
security aspects of the Windows binaries.  We are in effect leaving it
to the users to discover the problems and report them.

Re: Win32 GnuTLS DLL installer?

[Ted Zlatanov]

On Thu, 21 Sep 2017 22:08:42 +0100 phillip.lord@russet.org.uk (Phillip Lord) wrote: 

PL> To me, the sane "update" proceedure is "install a new version of
PL> emacs". If you want updating outside of that there are more holisitic
PL> solutions like chocolaty.

OK, let's focus on that then, and treat new DLL drops as a revision
increment, 25.3-2 for instance. Then the problem is reduced to building,
testing, and supporting a single version (which we already do). On the
user side, either an automated function or a manual "prominent button"
command will check that version and maybe optionally launch into a
package update.

On Thu, 21 Sep 2017 22:13:31 +0100 phillip.lord@russet.org.uk (Phillip Lord) wrote: 

PL> Ted Zlatanov <tzz@lifelogs.com> writes:

>> When a GitLab server is available, maybe we can set up a W32 build slave
>> to build and test these binaries.

PL> That would be a joy. At the moment, the binaries have minimal testing.

I also want to note that Docker images (ubuntu for instance) can run on
W32, so we can suggest that. That may work for some Emacs users and
reduce their dependence on W32.

PL> We don't have a good test suite, though, not for binaries. For example,
PL> if I don't add libXpm into the zip file, the tests will probably
PL> succeed, Emacs will run, but it will look very ropey.

OK, we'll plan to work on that.

On Fri, 22 Sep 2017 10:29:02 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

EZ> If we are willing to disregard the QA in these libraries for the
EZ> Windows users, then I submit that we should not consider seriously any
EZ> security aspects of the Windows binaries.  We are in effect leaving it
EZ> to the users to discover the problems and report them.

I don't think anyone is suggesting disregarding QA. We should work on
automating the QA process. Meanwhile we should do what we reasonably can
to support our users: focus on a single package and implement a way for
them to check that they are up to date.

(The Emacs model has been "users discover the problems and report them"
for many years. The notions of a test suite and CI are relatively very
recent and the test coverage is far from perfect. I think we'll only get
better at QA automation as our needs and capabilities evolve.)

Ted

Re: Win32 GnuTLS DLL installer?

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 1159 bytes --]

On Wed, Sep 20, 2017, at 10:15 PM, Phillip Lord wrote:
> On Wed, September 20, 2017 4:08 pm, Ted Zlatanov wrote:
>> It would be nice to have a Win32 installer for Emacs that
>> downloaded the>> GnuTLS DLLs.
>> 
>> 
>> Or maybe they can be packaged with the Win32 binaries. I don't
>> know the>> best solution.
> 
> 
> They are in
> https://ftp.gnu.org/gnu/emacs/windows/emacs-25-x86_64-deps.zip.
> 
> For emacs-26, I hope to start releasing two sets of zip files
> -- "emacs"> and "emacs-with-deps". The later will mean that you won't need to take> two
> packages, just unzip and go.
> 
> Actually, this will mean 4 packages -- 32 and 64 bit. Oh dear.
> 
> Phil
> 

Obviously this is not a GNU Level fix, but as a end-user I just wrote
some one-off batch files to automate the task for me, as part of my cross-
platform dotfiles-project.
I have scripts both for 32-bit and 64-bit Emacs. They can be found here,
for anyone interested, for any purpose:
https://github.com/josteink/machine-build/tree/master/tools

--
Regards
Jostein Kjønigsen

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjonigsen.net




[-- Attachment #2: Type: text/html, Size: 2351 bytes --]

Re: Win32 GnuTLS DLL installer?

[Ted Zlatanov]

On Wed, 27 Sep 2017 10:57:09 +0200 Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote: 

JK> Obviously this is not a GNU Level fix, but as a end-user I just wrote
JK> some one-off batch files to automate the task for me, as part of my cross-
JK> platform dotfiles-project.
JK> I have scripts both for 32-bit and 64-bit Emacs. They can be found here,
JK> for anyone interested, for any purpose:
JK> https://github.com/josteink/machine-build/tree/master/tools

Did you and Phillip Lord share this knowledge? It's not clear if he will
be able to use any of your work.

Ted

Re: Win32 GnuTLS DLL installer?

[Phillip Lord]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 27 Sep 2017 10:57:09 +0200 Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote: 
>
> JK> Obviously this is not a GNU Level fix, but as a end-user I just wrote
> JK> some one-off batch files to automate the task for me, as part of my cross-
> JK> platform dotfiles-project.
> JK> I have scripts both for 32-bit and 64-bit Emacs. They can be found here,
> JK> for anyone interested, for any purpose:
> JK> https://github.com/josteink/machine-build/tree/master/tools
>
> Did you and Phillip Lord share this knowledge? It's not clear if he will
> be able to use any of your work.

My new packages for Emacs-26 obviate the need for this script
really. The DLLs are prebundled now.

Phil