bug#51038: 27.2; ELPA certificate not trusted on Windows

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Michael Hoffman]

emacs.exe -Q --eval '(package-list-packages)' produces a *Network
Security Manager* buffer:

```
Certificate information
  Issued by:          R3
  Issued to:          CN=elpa.gnu.org
  Hostname:           elpa.gnu.org
  Public key:         RSA, signature: RSA-SHA256
  Session:            TLS1.3, key: ECDHE-RSA, cipher: AES-256-GCM, mac: AEAD
  Security level:     Medium
  Valid:              From 2021-09-28 to 2021-12-27

The TLS connection to elpa.gnu.org:443 is insecure
for the following reasons:

* certificate has expired
* certificate could not be verified
```

Output of `gnutls-cli.exe elpa.gnu.org:

```
|<1>| There was a non-CA certificate in the trusted list: OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft Corporation,CN=Microsoft Root Authority.
|<1>| There was a non-CA certificate in the trusted list: C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority.
|<1>| There was a non-CA certificate in the trusted list: CN=Root Agency.
Processed 55 CA certificate(s).
Resolving 'elpa.gnu.org:443'...
Connecting to '209.51.188.89:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
 - subject `CN=elpa.gnu.org', issuer `CN=R3,O=Let's Encrypt,C=US', serial 0x032e7afac8c8ff8acef5382c75dc16538637, RSA key 2048 bits, signed using RSA-SHA256, activated `2021-09-28 20:42:42 UTC', expires `2021-12-27 20:42:41 UTC', pin-sha256="WYj0qX4c/Xw7gDsCopUPyykUZoDxWda2RX3oSCAMTKE="
        Public Key ID:
                sha1:5641117962b98566f89ee43b392d5fa6a5c7e92d
                sha256:5988f4a97e1cfd7c3b803b02a2950fcb29146680f159d6b6457de848200c4ca1
        Public Key PIN:
                pin-sha256:WYj0qX4c/Xw7gDsCopUPyykUZoDxWda2RX3oSCAMTKE=

- Certificate[1] info:
 - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15 16:00:00 UTC', pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
- Certificate[2] info:
 - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US', issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30 18:14:03 UTC', pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate.
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
```

In `certlm.msc`, under "Certificates - Local Computer\Trusted Root
Certification Authorities\Certificates" there is a "DST Root CA X3"
certificate expiration 9/30/2021 serial number
44afb080d6a327ba893039862ef8406b.

There is also an "ISRG Root X1" certificate expiration 6/4/2035 serial number 008210cfb0d240e3594463e0bb63828b00.

It looks like GnuTLS is trying to check the certificate chain using the
DST Root CA X3 which has expired. The serial number and expiration for
the ISRG Root X1 in the certificates provided by elpa.gnu.org does not
match the one that Windows trusts.

Is this something that can be fixed on elpa.gnu.org? Something that I
need to fix in Windows?


In GNU Emacs 27.2 (build 1, x86_64-w64-mingw32)
 of 2021-03-26 built on CIRROCUMULUS
Repository revision: deef5efafb70f4b171265b896505b92b6eef24e6
Repository branch: HEAD
Windowing system distributor 'Microsoft Corp.', version 10.0.19043
System Description: Microsoft Windows 10 Home (v10.0.2009.19043.1237)

Configured using:
 'configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_US
  locale-coding-system: utf-8-unix

bug#51038: 27.2; ELPA certificate not trusted on Windows

[John Cummings]

Hi Michael, I'm just a user, but I've run into this bug recently and have been researching the options as they relate to Emacs. I believe that this is what you should expect if you are using gnutls 3.6.12 and you have the expired X3 root cert in your trust store. As you're seeing, that version of gnutls treats the chain as expired because of the expired root, even though it could validate it due to the alternate path leading to the ISRG root. I confirmed both of those on my Emacs 27.2 Windows installation from the builds kindly published by GNU, which I assume you are using. Since this is a self-contained build not living in a package management system, I don't BELIEVE there is any good way to fix the root cause of the problem on your system without rebuilding Emacs with gnutls >= 3.6.14, so I'm not sure if the maintainers will close this, or slot it for the next Windows build that gets published.

But hopefully this is something you can address on your side for now. Since this is expected behavior, the least invasive thing to do is probably to decide to trust that certificate (a)lways, assuming you are confident in its identity. I am personally confident in that, because I verified that the key checksums Emacs is reporting do belong to elpa.gnu.org. You don't have to take my word for that; you can download the cert in a browser that you trust (which probably will not be experiencing this problem), and then dump the public key info with something like "certtool --infile=elpa-gnu-org.pem --pubkey". I believe that certtool is bundled in that Windows installation.

I was also able to bypass this problem by removing that expired X3 root cert from my list of trusted roots in Windows, but it seems risky and unnecessary compared with the previous option. So I'm not recommending that, just noting that it seems to work for me.

This issue could be addressed on the server side as well, but some services are choosing to leave this chain with the expired root in place. There are valid reasons to do this, and correct clients (like gnutls 3.6.14) should be able to handle it, but I don't know the specifics on why GNU has chosen to leave it so far.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Lars Ingebrigtsen]

John Cummings <john@rootabega.net> writes:

> I believe that this is what you should expect if you are using gnutls
> 3.6.12 and you have the expired X3 root cert in your trust store.

Yup.  So this isn't a problem on Savannah.  Quoting:

----
From: Bob Proulx <INVALID.NOREPLY@gnu.org>
Subject: Certificate Expiration Event September 2021

On September 30, 2021, as planned the DST Root CA X3 cross-sign has expired
for the Let's Encrypt trust chain.  That was a normal and planned event. 
However coupled with a verification error in the code of libraries
authenticating certificates it caused some clients that have not been updated
to fixed versions to have problems validating certificates.

If you are experiencing invalid certificate chain problems with Let's Encrypt
certificates (not a Savannah problem) then please upgrade your client to the
latest security patches for your system.  Please reference these resources as
to upstream information and discussion about the issue.

* https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
* https://community.letsencrypt.org/t/production-chain-changes/150739/4
* https://letsencrypt.org/docs/certificate-compatibility/
* https://letsencrypt.org/certificates/
* https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/
----

So I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51038: 27.2; ELPA certificate not trusted on Windows

[John Cummings]

Lars Ingebrigtsen <larsi@gnus.org> wrote:

> John Cummings john@rootabega.net writes:
>
> > I believe that this is what you should expect if you are using gnutls
> > 3.6.12 and you have the expired X3 root cert in your trust store.
> Yup. So this isn't a problem on Savannah. Quoting:
>
> -----------------------------------------------------
>
> From: Bob Proulx INVALID.NOREPLY@gnu.org
> certificates (not a Savannah problem) then please upgrade your client to the
> latest security patches for your system.

Is there a recommended way to do that for the Windows builds of Emacs published to ftp.gnu.org?

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Eli Zaretskii]

> Date: Wed, 06 Oct 2021 10:54:01 +0000
> From: John Cummings <john@rootabega.net>
> Cc: Michael Hoffman <emacs-hoffman@snkmail.com>
> 
> > > I believe that this is what you should expect if you are using gnutls
> > > 3.6.12 and you have the expired X3 root cert in your trust store.
> > Yup. So this isn't a problem on Savannah. Quoting:
> >
> > -----------------------------------------------------
> >
> > From: Bob Proulx INVALID.NOREPLY@gnu.org
> > certificates (not a Savannah problem) then please upgrade your client to the
> > latest security patches for your system.
> 
> Is there a recommended way to do that for the Windows builds of Emacs published to ftp.gnu.org?

I don't understand: you want to do what for that build?

bug#51038: 27.2; ELPA certificate not trusted on Windows

[John Cummings]

Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 06 Oct 2021 10:54:01 +0000
>> From: John Cummings <john@rootabega.net>
>> Cc: Michael Hoffman <emacs-hoffman@snkmail.com>
>>
>> > > I believe that this is what you should expect if you are using gnutls
>> > > 3.6.12 and you have the expired X3 root cert in your trust store.
>> > Yup. So this isn't a problem on Savannah. Quoting:
>> >
>> > -----------------------------------------------------
>> >
>> > From: Bob Proulx INVALID.NOREPLY@gnu.org
>> > certificates (not a Savannah problem) then please upgrade your client to the
>> > latest security patches for your system.
>>
>> Is there a recommended way to do that for the Windows builds of Emacs published to ftp.gnu.org?
>
>I don't understand: you want to do what for that build?

Upgrade the client -- emacs and gnutls -- to work with this trust chain.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Eli Zaretskii]

> Date: Wed, 06 Oct 2021 13:12:14 +0000
> From: John Cummings <john@rootabega.net>
> Cc: 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com
> 
> >I don't understand: you want to do what for that build?
> 
> Upgrade the client -- emacs and gnutls -- to work with this trust chain.

That's not how this stuff works on MS-Windows.  There, the
certificates are stored by the system, and GnuTLS (and Emacs) just use
what the system stores.  See gnutls_certificate_set_x509_system_trust.

So you need to update your Windows system, and then everything will
work.

Of course, you can install a cert bundle from someplace and tell
GnuTLS to use it.  But I'm not sure this is easy on Windows, or that
it would override the system's trust store, or even where to download
that from.  And in any case, the fix is again on your system, not in
Emacs or in GnuTLS: those don't come with any certificate DB, at least
not AFAIK.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[John Cummings]

Eli Zaretskii <eliz@gnu.org> wrote:

>> Date: Wed, 06 Oct 2021 13:12:14 +0000
>> From: John Cummings <john@rootabega.net>
>> Cc: 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com
>>
>> >I don't understand: you want to do what for that build?
>>
>> Upgrade the client -- emacs and gnutls -- to work with this trust chain.
>
> That's not how this stuff works on MS-Windows.

That's how it works on any system running gnutls 3.6.12, no? The bug
in gnutls is fixed in 3.6.14.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Michael Hoffman]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

Thanks John for the detailed analysis. Windows Update reports my system is
up to date. As I understand it:

1. My certificate store is valid.
2. Savannah's HTTPS responses are valid.
3. Emacs 27.2 per se's behavior is valid.
4. GnuTLS 3.6.12 has a bug that produces incorrect results even in presence
of 1-2.
5. Emacs 27.2 Windows binaries from ftp.gnu.org include GnuTLS 3.6.12,
which has this bug.

The Emacs 28 pretest Windows binaries from earlier in the year include GnuTLS
3.6.15. I hope this means everything will work as expected on the final
Emacs 28.1 Windows binaries.

[-- Attachment #2: Type: text/html, Size: 928 bytes --]

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Eli Zaretskii]

> Date: Wed, 06 Oct 2021 13:39:50 +0000
> From: John Cummings <john@rootabega.net>
> Cc: larsi@gnus.org, 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com
> 
> > That's not how this stuff works on MS-Windows.
> 
> That's how it works on any system running gnutls 3.6.12, no? The bug
> in gnutls is fixed in 3.6.14.

Maybe we aren't talking about the same bug, then.  AFAIU, the problem
is supposed to be solved by updating the cert bundle, isn't that so?

If the bug is in GnuTLS, then simply install a newer one from the
MSYS2 site, and that's it.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[John Cummings]

Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 06 Oct 2021 13:39:50 +0000
>> From: John Cummings <john@rootabega.net>
>> Cc: larsi@gnus.org, 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com
>>
>> > That's not how this stuff works on MS-Windows.
>>
>> That's how it works on any system running gnutls 3.6.12, no? The bug
>> in gnutls is fixed in 3.6.14.

> Maybe we aren't talking about the same bug, then.  AFAIU, the problem
> is supposed to be solved by updating the cert bundle, isn't that so?

In my understanding, the root cause is that GnuTLS focuses on the
expired root without considering alternate paths, so removing the
expired root hides the behavior, but GnuTLS would still need fixing.

> If the bug is in GnuTLS, then simply install a newer one from the
> MSYS2 site, and that's it.

That makes sense to me as one possible way to correct this. It seems
like we all agree that the 27.2 Windows build on ftp.gnu.org has this
"potential for undesirable behavior" (if the term "bug" doesn't sit
right with anyone.) I thought this bug report would end up serving
to:

1. acknowledge the behavior in that specific binary

2. list fixes/workarounds like updating GnuTLS individually,
   or modifying the system trust store

3. communicate that this behavior will no longer happen in
   the version 28 binaries (once released), for those who might not
   be in a position to update GnuTLS independently, or would
   rather wait for an updated binary with deps.

I understand that the Windows binaries are a volunteer courtesy, so if
nothing else, I think users of that binary would benefit from some
formal thing telling them that this behavior exists and will
eventually be changed. Hopefully that's already accomplished, and
people will just find this bug if they search, and understand the
situation with respect to the v 28 Windows binaries.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Ioannis Kappas]

Hi all,

doublep/eldev (the Elisp Development Tool) is just an example of a
project  that has been affected by this issue and has taken some time
and serious effort to figure out what was going wrong:
https://github.com/doublep/eldev/issues/55. The CI running the eldev
test suit on GitHub Windows 2019 servers, which involved downloading
packages from MELPA, suddenly started to fail one day when connecting
to stable.melpa.org. The same tests passed on Linux/MacOs builds. I am
sure there are many more other such instances, either projects or just
users that are affected by it, and are perplexed with the current
situation without having knowledge of the root cause.

May I please argue that this should be at least acknowledged as an
important issue with the latest official GNU emacs 27.2 binary
MS-Windows release as advertised in the `GNU Emacs Download & Install
page' @  https://www.gnu.org/software/emacs/download.html, under
`Nonfree systems`->Windows:

"""GNU Emacs for Windows can be downloaded from a nearby GNU mirror;
or the main GNU FTP server"""

Where `GNU mirror points` to
http://ftp.gnu.org/gnu/emacs/windows/emacs-27/, with the affected
emacs 27.2 releases dated on 2021-Mar-31.

The issue is that the *latest official Gnu Emacs windows binary
releases*, as of today, at the official GNU Emacs download site are
*bundled* with gnutls-3.6.12 which is susceptible to GnuTLS bug#1008
(titled as Handle expiration of AddTrust root certificate (urgent) --
https://gitlab.com/gnutls/gnutls/-/issues/1008) which refuses
connections to sites with valid certificates whose issuer consist of
dual certificates of which one has expired but the other is
not-expired i.e. valid. As such, the official precompiled Emacs 27.2
Windows binaries cannot connect to these sites, which severely
compromising Emacs functionality, with preventing Emacs connecting to
package archives such as ELPA or MELPA being the most prevailing
example.

Thus, I advocate, that the latest official precompiled Gnu Emacs
MS-Windows binaries have a serious issue (caused by a bug in the
GnuTLS version they are bundled with), that either needs to be
addressed or a workaround needs to be suggested somewhere in the
download/install instructions.

For completion I list the available options discussed in this thread/I
can think of with any disadvantages I can think of:

1. Fix: Release new precompiled Emacs 27.2 binary versions to the
official site bundled with a GnuTLS version that has GnuTLS#1008 fix,
i.e. with version >= 3.6.14 (is this likely to be a release
nightmare?)
2. Fix: Wait until the next release (I believe 28.x release is around
the corner?). This leaves Emacs users which rely on the latest
official build vulnerable; i.e. users that follow the official
instructions and don't know what MSYS2 is or how to use it or can't be
bothered -- this is probably the majority of nontechnical users -- or
users in systems behind corporate firewalls that do not permit install
of third party tools msys2/chocolately/scoop, or users in remote
servers with preinstalled version of latest emacs version -- for
example GitHub windows 2019 build/test farms.
3. Work around: Document the issue somewhere that the a prospective
user can't miss (e.g. official download page or the readme document
alongside the binaries, anything else?), with workarounds being
3.1 Update windows certificate store to remove expired certificate as
mentioned in this thread (not sure how this would work, how do you
users find the list of the ones that expired? Does it require special
permission to remove certs? I suppose `Let's encrypt' issuers
certificates are not the only one affected, they may be more either
now or down the line).
3.2 use MSYS2 to build (pickup?) a 27.2 version with the latest GnuTLS
lib (or chocolatey, or scoop perhaps if such version exist there).
Though user might not have the technical background to do so or the
host is restricted in respect to the tools that can be installed
(systems behind corporate firewalls) or the target system is a server
with limited access as to the choice of tools that can be installed
(e.g. custom build Windows 2019 github server farms).
4 Work around: the same as #3 but without updating instructions about
the problem or how to fix it. Leaves users who rely on the latest
official releases without knowledge of this issue in the most
vulnerable and perplexed for them situation.

Thank you

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Eli Zaretskii]

> From: Ioannis Kappas <ioannis.kappas@gmail.com>
> Date: Sun, 24 Oct 2021 17:49:26 +0100
> Cc: 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com, 
> 	Lars Ingebrigtsen <larsi@gnus.org>
> 
> Thus, I advocate, that the latest official precompiled Gnu Emacs
> MS-Windows binaries have a serious issue (caused by a bug in the
> GnuTLS version they are bundled with), that either needs to be
> addressed or a workaround needs to be suggested somewhere in the
> download/install instructions.

AFAIU, this issue is not with Emacs, it's with GnuTLS.  So all you
need is download and install a newer GnuTLS, where this problem was
fixed, in place of the old one.  Emacs should then work with that
GnuTLS; there's no need to rebuild Emacs itself.

If you already tried that and it didn't work, please tell the details.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Ioannis Kappas]

Hi Eli,

On Sun, Oct 24, 2021 at 6:11 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > Thus, I advocate, that the latest official precompiled Gnu Emacs
> > MS-Windows binaries have a serious issue (caused by a bug in the
> > GnuTLS version they are bundled with), that either needs to be
> > addressed or a workaround needs to be suggested somewhere in the
> > download/install instructions.
>
> AFAIU, this issue is not with Emacs, it's with GnuTLS.  So all you
> need is download and install a newer GnuTLS, where this problem was
> fixed, in place of the old one.  Emacs should then work with that
> GnuTLS; there's no need to rebuild Emacs itself.
>
> If you already tried that and it didn't work, please tell the details.

(apologies for being pedantic here, just want tom make sure that any
difference in opinion become clear)

before going into the details of a workaround, my argument is that
this is an issue with the precompiled binaries of the latest official
Gnu Emacs release at the official ftp site. If a user or process
installs today these binaries on their system, Emacs will not work to
its full potential. Furthermore, the user will not be aware why the
connection to the elpa archive fails nor of a potential work around. I
consider this to be a major issue with the precompiled binaries
prepared by the Gnu Emacs projects, that they don't work out of the
box and likely to leave the user/system in a perplexed/volnurable
state.

I believe you are saying that there is no issue with the latest
official precompiled Gnu Emacs Windows release (say at
http://ftp.gnu.org/gnu/emacs/windows/emacs-27/emacs-27.2-x86_64.zip),
because the error is coming from libgnu-3.6.12, a library that Emacs
depends on, and not from the Emacs code.

May I point out that libgnu-2.6.12 ships in emacs-27.2-x86_64.zip
under bin/libgnutls-30.dll, and thus the responsibility to the
maintainer of the package to fix any shortfalls IMHO? Currently the
official instructions to install the latest Gnu Emacs release from the
precompiled binaries from the official ftp site, install a version of
Emacs which is impaired, and wont work to its full potential out of
the box for any user.  We need to either fix this so it works out of
the box, provide official instructions how to work around it, or
provide an official note that this is broken. Letting users being
unaware and thus vulnerable to the current behaviour IMHO is
suboptimal.

---

With regards to the suggested workaround, on my Windows machine

1. I've downloaded and unpacked
http://ftp.gnu.org/gnu/emacs/windows/emacs-27/emacs-27.2-x86_64.zip to
a local directory.
2. Looking for the GnuTLS precompiled version for windows, I landed on
this page: https://www.gnutls.org/download.html
2.1 There is a latest w64 version on gitlab link at
https://gitlab.com/gnutls/gnutls/builds/artifacts/3.7.2/download?job=MinGW64.DLLs
that redirects to a 404.
2.1.1 Trying to find the artifacts by going to
https://gitlab.com/gnutls/gnutls -> CI/CD -> Pipelines -> click on
pipeline ID (in my case
#392652428)->Jobs->mingw64/archive->Browser->Win64-build->bin/libgnutls-30.dll->Download
(quite a mouthful) and replace libgnutls-30.dll with it works.

Which i find it a bit too involved, especially for new users
regardless even if they are magically aware of the root issue?

Thanks!

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Lars Ingebrigtsen]

Ioannis Kappas <ioannis.kappas@gmail.com> writes:

> before going into the details of a workaround, my argument is that
> this is an issue with the precompiled binaries of the latest official
> Gnu Emacs release at the official ftp site.

Yes, if somebody could build a new version of those (with an updated
version of the gnutls libraries), that'd be nice.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Eli Zaretskii]

tags 51038 notabug
thanks

> From: Ioannis Kappas <ioannis.kappas@gmail.com>
> Date: Sun, 24 Oct 2021 19:21:00 +0100
> Cc: john@rootabega.net, 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com, 
> 	Lars Ingebrigtsen <larsi@gnus.org>
> 
> (apologies for being pedantic here, just want tom make sure that any
> difference in opinion become clear)

I wasn't aware that we are having differences of opinions here.

> before going into the details of a workaround, my argument is that
> this is an issue with the precompiled binaries of the latest official
> Gnu Emacs release at the official ftp site. If a user or process
> installs today these binaries on their system, Emacs will not work to
> its full potential. Furthermore, the user will not be aware why the
> connection to the elpa archive fails nor of a potential work around. I
> consider this to be a major issue with the precompiled binaries
> prepared by the Gnu Emacs projects, that they don't work out of the
> box and likely to leave the user/system in a perplexed/volnurable
> state.

That is true, but users who download precompiled binaries are at the
mercy of whoever prepared the package from the get-go, so this danger
is not new, it is inherent to this way of installing Emacs.  People
who want to be completely in control should compile Emacs by
themselves.  We have instructions for that in nt/INSTALL.W64.

> May I point out that libgnu-2.6.12 ships in emacs-27.2-x86_64.zip
> under bin/libgnutls-30.dll, and thus the responsibility to the
> maintainer of the package to fix any shortfalls IMHO?

We don't have a maintainer at this time.  This was (and is) a
volunteer project, and the volunteer who produced that bundle stepped
down.  If you'd like to replace him, I'm sure this will be very
welcome.  Or maybe someone else will soon.

> Currently the
> official instructions to install the latest Gnu Emacs release from the
> precompiled binaries from the official ftp site, install a version of
> Emacs which is impaired, and wont work to its full potential out of
> the box for any user.  We need to either fix this so it works out of
> the box, provide official instructions how to work around it, or
> provide an official note that this is broken. Letting users being
> unaware and thus vulnerable to the current behaviour IMHO is
> suboptimal.

There's a problem with the "we" part here.  There's also a problem
with providing instructions, because the fine details depend on what
is already installed on the end-user's system.  It's hard to provide a
cookbook here.

> With regards to the suggested workaround

It isn't a workaround, it's THE solution.

> 1. I've downloaded and unpacked
> http://ftp.gnu.org/gnu/emacs/windows/emacs-27/emacs-27.2-x86_64.zip to
> a local directory.
> 2. Looking for the GnuTLS precompiled version for windows, I landed on
> this page: https://www.gnutls.org/download.html
> 2.1 There is a latest w64 version on gitlab link at
> https://gitlab.com/gnutls/gnutls/builds/artifacts/3.7.2/download?job=MinGW64.DLLs
> that redirects to a 404.

The correct place to update GnuTLS is from the MSYS2 project, which is
where all the optional DLLs in the binary bundle come from.  The URL
is in nt/INSTALL.W64; start by installing pacman, and then fetch the
latest mingw64 libgnutls DLLs.  (I myself don't use that, so
unfortunately I cannot give you more details, but perhaps someone else
here will.)

Alternatively, I believe you can tell the Emacs NSM, once, to trust
ELPA regardless of the certificate, and then it will work henceforth.
(This _is_ a workaround.)

In any case, this is not a bug in Emacs.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Ioannis Kappas]

On Sun, Oct 24, 2021 at 7:50 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Ioannis Kappas <ioannis.kappas@gmail.com>
> > Date: Sun, 24 Oct 2021 19:21:00 +0100
> > Cc: john@rootabega.net, 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com,
> >       Lars Ingebrigtsen <larsi@gnus.org>
> >
> > (apologies for being pedantic here, just want tom make sure that any
> > difference in opinion become clear)
>
> I wasn't aware that we are having differences of opinions here.

Great, I think we came to an understanding

1. Issue is  not with Emacs.
2. Issue with the latest official precompiled binaries of Gnu Emacs
for MS-Windows, caused by the bundled GnuTLS library version.


> > before going into the details of a workaround, my argument is that
> > this is an issue with the precompiled binaries of the latest official
> > Gnu Emacs release at the official ftp site. If a user or process
> > installs today these binaries on their system, Emacs will not work to
> > its full potential. Furthermore, the user will not be aware why the
> > connection to the elpa archive fails nor of a potential work around. I
> > consider this to be a major issue with the precompiled binaries
> > prepared by the Gnu Emacs projects, that they don't work out of the
> > box and likely to leave the user/system in a perplexed/volnurable
> > state.
>
> That is true, but users who download precompiled binaries are at the
> mercy of whoever prepared the package from the get-go, so this danger
> is not new, it is inherent to this way of installing Emacs.  People
> who want to be completely in control should compile Emacs by
> themselves.  We have instructions for that in nt/INSTALL.W64.

May I disagree with this that there is nothing to suggest that in the
the official download page @
https://www.gnu.org/software/emacs/download.html, under Nonfree
systems/Windows:

"""
Windows

GNU Emacs for Windows can be downloaded from a nearby GNU mirror; or
the main GNU FTP server.
Unzip the zip file preserving the directory structure, and run
bin\runemacs.exe. Alternatively, create a desktop shortcut to
bin\runemacs.exe, and start Emacs by double-clicking on that
shortcut's icon.

The Windows binaries are signed by Phillip Lord 8E64 B119 FE4B AC58
C767 D5EC E095 C1A6 3FB1 EAD2.
"""

If this is the official position, IMHO it should be clearly stated
somewhere obvious (unless I missed it). Otherwise people old or new to
emacs think these precompiled binaries are officially supported by the
project maintainers and should work out of the box.

> > May I point out that libgnu-2.6.12 ships in emacs-27.2-x86_64.zip
> > under bin/libgnutls-30.dll, and thus the responsibility to the
> > maintainer of the package to fix any shortfalls IMHO?
>
> We don't have a maintainer at this time.  This was (and is) a
> volunteer project, and the volunteer who produced that bundle stepped
> down.  If you'd like to replace him, I'm sure this will be very
> welcome.  Or maybe someone else will soon.

I could possibly assist if needs be. I assume this is based on
trusting the person creating the package rather than having an
automated build process in place.

> > Currently the
> > official instructions to install the latest Gnu Emacs release from the
> > precompiled binaries from the official ftp site, install a version of
> > Emacs which is impaired, and wont work to its full potential out of
> > the box for any user.  We need to either fix this so it works out of
> > the box, provide official instructions how to work around it, or
> > provide an official note that this is broken. Letting users being
> > unaware and thus vulnerable to the current behaviour IMHO is
> > suboptimal.
>
> There's a problem with the "we" part here.  There's also a problem
> with providing instructions, because the fine details depend on what
> is already installed on the end-user's system.  It's hard to provide a
> cookbook here.

My experience with the precompiled binaries zip file is well self
contained and does not depend on anything outside of it, other than
the windows kernel.

> > With regards to the suggested workaround
>
> It isn't a workaround, it's THE solution.

OK, there is a slight difference of opinion here. The solution for me
is to update the precompiled binaries with a recent GnuTLS version on
the official download site.  Having the user to install MSYS2 and
locate the dll (or download the latest version from the GnuTLS CI) as
to overwrite the a single dll in the official precompiled binary,
sounds like a work around to me.

> > 1. I've downloaded and unpacked
> > http://ftp.gnu.org/gnu/emacs/windows/emacs-27/emacs-27.2-x86_64.zip to
> > a local directory.
> > 2. Looking for the GnuTLS precompiled version for windows, I landed on
> > this page: https://www.gnutls.org/download.html
> > 2.1 There is a latest w64 version on gitlab link at
> > https://gitlab.com/gnutls/gnutls/builds/artifacts/3.7.2/download?job=MinGW64.DLLs
> > that redirects to a 404.
>
> The correct place to update GnuTLS is from the MSYS2 project, which is
> where all the optional DLLs in the binary bundle come from.  The URL
> is in nt/INSTALL.W64; start by installing pacman, and then fetch the
> latest mingw64 libgnutls DLLs.  (I myself don't use that, so
> unfortunately I cannot give you more details, but perhaps someone else
> here will.)

Yeah, I've tested this to work too. I was trying to follow up what I
thought was your suggestion earlier and whence the instructions above.

> Alternatively, I believe you can tell the Emacs NSM, once, to trust
> ELPA regardless of the certificate, and then it will work henceforth.
> (This _is_ a workaround.)

You do get a prompt when `packages' try to connect in the GUI, but in
the batch mode (as in the Eldev case, where the error happens on a
cloud server somewher in GitHub without the user input) you don't,
though it should be possible to disable programmatically as a possible
work around indeed.

> In any case, this is not a bug in Emacs.

Agreed, it is an issue with the latest official precompiled MS-Windows binaries.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Eli Zaretskii]

> From: Ioannis Kappas <ioannis.kappas@gmail.com>
> Date: Sun, 24 Oct 2021 21:30:09 +0100
> Cc: john@rootabega.net, 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com, 
> 	Lars Ingebrigtsen <larsi@gnus.org>
> 
> > That is true, but users who download precompiled binaries are at the
> > mercy of whoever prepared the package from the get-go, so this danger
> > is not new, it is inherent to this way of installing Emacs.  People
> > who want to be completely in control should compile Emacs by
> > themselves.  We have instructions for that in nt/INSTALL.W64.
> 
> May I disagree with this that there is nothing to suggest that in the
> the official download page @
> https://www.gnu.org/software/emacs/download.html, under Nonfree
> systems/Windows:
> 
> """
> Windows
> 
> GNU Emacs for Windows can be downloaded from a nearby GNU mirror; or
> the main GNU FTP server.
> Unzip the zip file preserving the directory structure, and run
> bin\runemacs.exe. Alternatively, create a desktop shortcut to
> bin\runemacs.exe, and start Emacs by double-clicking on that
> shortcut's icon.
> 
> The Windows binaries are signed by Phillip Lord 8E64 B119 FE4B AC58
> C767 D5EC E095 C1A6 3FB1 EAD2.
> """
> 
> If this is the official position, IMHO it should be clearly stated
> somewhere obvious (unless I missed it). Otherwise people old or new to
> emacs think these precompiled binaries are officially supported by the
> project maintainers and should work out of the box.

You are reading too much into that text.  It doesn't say anywhere that
these binaries are "official', nor even that they are endorsed or
blessed by the project.  I don't think it's reasonable to expect us to
have a disclaimer near any binary distribution of Emacs saying it
isn't "official".  There are more sites out there which distribute
precompiled binaries of Emacs.

> > There's a problem with the "we" part here.  There's also a problem
> > with providing instructions, because the fine details depend on what
> > is already installed on the end-user's system.  It's hard to provide a
> > cookbook here.
> 
> My experience with the precompiled binaries zip file is well self
> contained and does not depend on anything outside of it, other than
> the windows kernel.

I wasn't talking about dependencies, I was talking about additional
Emacs-related software that could be installed on the user's machine,
and could affect the detailed instructions.  For example, some of
those additional programs could require the old GnuTLS, so we cannot
easily say "replace with the new one".

> OK, there is a slight difference of opinion here. The solution for me
> is to update the precompiled binaries with a recent GnuTLS version on
> the official download site.

The only official download for Emacs is the source distribution.  That
is the only distribution that's under the direct responsibility of the
project.

> > Alternatively, I believe you can tell the Emacs NSM, once, to trust
> > ELPA regardless of the certificate, and then it will work henceforth.
> > (This _is_ a workaround.)
> 
> You do get a prompt when `packages' try to connect in the GUI, but in
> the batch mode (as in the Eldev case, where the error happens on a
> cloud server somewher in GitHub without the user input) you don't,
> though it should be possible to disable programmatically as a possible
> work around indeed.

If you answer "allow" for that single prompt, telling the NSM to
always trust ELPA, you won't need to answer any more questions, and I
believe the batch operation will also work.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Ioannis Kappas]

On Mon, Oct 25, 2021 at 12:48 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Ioannis Kappas <ioannis.kappas@gmail.com>
> > Date: Sun, 24 Oct 2021 21:30:09 +0100
> > Cc: john@rootabega.net, 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com,
> >       Lars Ingebrigtsen <larsi@gnus.org>
> >

> > If this is the official position, IMHO it should be clearly stated
> > somewhere obvious (unless I missed it). Otherwise people old or new to
> > emacs think these precompiled binaries are officially supported by the
> > project maintainers and should work out of the box.
>
> You are reading too much into that text.  It doesn't say anywhere that
> these binaries are "official', nor even that they are endorsed or
> blessed by the project.  I don't think it's reasonable to expect us to
> have a disclaimer near any binary distribution of Emacs saying it
> isn't "official".  There are more sites out there which distribute
> precompiled binaries of Emacs.

I believe the perception for the majority of users new or old to Emacs
is that these are the official binaries. As of a random example, the
Emacs Wiki @ https://www.emacswiki.org/emacs/MsWindowsInstallation
reads (the *** are mine):

""" Guidelines for installing Emacs on MS Windows

To install the ***official*** stable binaries:

  Visit https://ftp.gnu.org/gnu/emacs/windows/emacs-27/ (or
https://ftpmirror.gnu.org/emacs/windows/emacs-27/ to use a nearby
mirror). Download the last zip-file ending in x86_64.zip for 64 bit or
i686.zip for 32 bit listed (currently emacs-27.1-x86_64.zip and
emacs-27.1-i686.zip, respectively). You might also want to read the
README file at the same site (not the one inside the zip-file). Once
the zip-file is downloaded, open it using Explorer (slow) or 7zip
(faster) and extract all the files into a directory of your choice
(e.g. c:\packages\emacs-27.1).""

Now, you may argue, anyone can right anything on these websites, but I
am just trying to give out the sentiment what the people think about
these precompiled binaries. I would personally not have thought these
were not official until we had this discussion. I think this
misconception should be addressed somehow, otherwise users might be
left perplexed or unable to use Emacs on Windows to its full
potential. That's my personal opinion btw, you don't have to agree
with it. You are the maintainers after all and your decisions are most
respected.

> > > There's a problem with the "we" part here.  There's also a problem
> > > with providing instructions, because the fine details depend on what
> > > is already installed on the end-user's system.  It's hard to provide a
> > > cookbook here.
> >
> > My experience with the precompiled binaries zip file is well self
> > contained and does not depend on anything outside of it, other than
> > the windows kernel.
>
> I wasn't talking about dependencies, I was talking about additional
> Emacs-related software that could be installed on the user's machine,
> and could affect the detailed instructions.  For example, some of
> those additional programs could require the old GnuTLS, so we cannot
> easily say "replace with the new one".
> > OK, there is a slight difference of opinion here. The solution for me
> > is to update the precompiled binaries with a recent GnuTLS version on
> > the official download site.
>
> The only official download for Emacs is the source distribution.  That
> is the only distribution that's under the direct responsibility of the
> project.

IMHO we have to make this clear somehow. The believe in my opinion is
that people are seeing these binaries as official.

> > > Alternatively, I believe you can tell the Emacs NSM, once, to trust
> > > ELPA regardless of the certificate, and then it will work henceforth.
> > > (This _is_ a workaround.)
> >
> > You do get a prompt when `packages' try to connect in the GUI, but in
> > the batch mode (as in the Eldev case, where the error happens on a
> > cloud server somewher in GitHub without the user input) you don't,
> > though it should be possible to disable programmatically as a possible
> > work around indeed.
>
> If you answer "allow" for that single prompt, telling the NSM to
> always trust ELPA, you won't need to answer any more questions, and I
> believe the batch operation will also work.

In our case, the failing Emacs process is run on a Windows 2019 server
noninteractively somewhere on the GitHub cloud, with no user
intervention so there is no prompt to see the prompt. I figured out
though a hack based on this, It should be possible to include in the
testing script. Thanks for mentioning it.

bug#51038: 27.2; ELPA certificate not trusted on Windows

[Ioannis Kappas]

On Mon, Oct 25, 2021 at 6:18 PM Ioannis Kappas <ioannis.kappas@gmail.com> wrote:
>
> On Mon, Oct 25, 2021 at 12:48 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Ioannis Kappas <ioannis.kappas@gmail.com>
> > > Date: Sun, 24 Oct 2021 21:30:09 +0100
> > > Cc: john@rootabega.net, 51038@debbugs.gnu.org, emacs-hoffman@snkmail.com,
> > >       Lars Ingebrigtsen <larsi@gnus.org>
> > >
>
> > > If this is the official position, IMHO it should be clearly stated
> > > somewhere obvious (unless I missed it). Otherwise people old or new to
> > > emacs think these precompiled binaries are officially supported by the
> > > project maintainers and should work out of the box.
> >
> > You are reading too much into that text.  It doesn't say anywhere that
> > these binaries are "official', nor even that they are endorsed or
> > blessed by the project.  I don't think it's reasonable to expect us to
> > have a disclaimer near any binary distribution of Emacs saying it
> > isn't "official".  There are more sites out there which distribute
> > precompiled binaries of Emacs.
>
> I believe the perception for the majority of users new or old to Emacs
> is that these are the official binaries. As of a random example, the
> Emacs Wiki @ https://www.emacswiki.org/emacs/MsWindowsInstallation
> reads (the *** are mine):
>
> """ Guidelines for installing Emacs on MS Windows
>
> To install the ***official*** stable binaries:

Hi again, here is some more evidence that these prebuild binaries are
widely considered to be official GNU packages. They are picked up by
the two major MS-Windows package managers I am aware of, and thus
almost everyone is affected when something goes wrong:

Chocolatey: https://community.chocolatey.org/packages/Emacs

Scoop: https://github.com/ScoopInstaller/Extras/blob/master/bucket/emacs.json

(and what Eldev used as an installer in the GitHub action:
https://github.com/purcell/setup-emacs)

I would like to stress again that IMO it should be made clear
somewhere prominent that these precompiled binaries published on the
GNU ftp site are unofficial, unmaintained,  unsupported binary
packages (as I believe is Eli's position) and people or processes
should not rely  on them, but rather build their own packages from
source (which is a non-trivial task for many people and requires
access to MSYS2). I don't believe the package managers are wrong here
for picking up the binaries from the official GNU site. It is the
inherited belief that these are official supported binaries that is
the issue IMHO.

Sorry for bringing this up again, but I do believe this is a major
issue which requires addressing affecting many people and processes
out there.

Thanks