bug#35739: Bad signature from GNU ELPA

[Stefan Monnier <monnier@iro.umontreal.ca>]

>> It solves the problem by refraining from decoding until we know
>> positively that it needs to happen.
> You refrain from decoding what?

The files we download.

> E.g., Lisp files must be in UTF-8, right?

At first we don't care: we gets files (tarballs, GPG signatures, Elisp
files, ...) and while some of them may need decoding later on, not all
do.  And for purposes of signature checking, for some of those files we
need to get the exact original sequence of bytes, which is easier to get
if we only decode *after* signature checking rather than before.
For this reason we don't want to let URL do the decoding for us.

It's true that for "simple packages" made of a single .el file, after
signature checking we should maybe decode them as utf-8.  The main thing
we do with those is to save them as files and for that we don't
need decoding.  I haven't checked whether we do anything else
significant with those undecoded .el buffers, so maybe I missed an
explicit decode somewhere in there.

> And what about the descriptions we show in lisp-packages?

Not sure what you mean by that (I already mentioned the *-readme.txt
files which we do decode explicitly now).

> Or maybe I don't really understand why we need to decode _anything_,
> since we just download a .tar archive, right?

There's more than just tarballs.

>> Yes, this part should definitely not be in emacs-26.
> I'm actually asking why it should be on master.

It seemed like a simple way to provide this new functionality.
The functionality is needed at least by package.el, and I see no reason
why it should be the only client of URL that needs this functionality.

>> the change in url-insert only affects the case where the HTTP
>> headers returned by the server specify a particular "charset", which
>> is not the case when downloading .tar and .el files from
>> elpa.gnu.org AFAICT
> Why not?

For tarballs, since it's not a text/* format, it wouldn't make much
sense to specify a charset.  For Elisp files, it might just be a happy
accident of the configuration of the HTTP server, but my impression is
that nowadays it's considered a bad idea to rely on the HTTP headers to
tell you about the encoding (instead, the data contents should specify
its own encoding), so it's only useful for text/plain files.

> isn't that dangerous?

Dangerous?  definitely not.  All it means is that if you try to view
something like https://elpa.gnu.org/packages/xclip-1.8.el in your
browser there's a possibility that it will not display the non-ASCII
characters properly.  Of course, most browsers I know won't display
the file at all anyway (they ask you where to save it instead because
they don't know what else to do with text/x-emacs-lisp).


        Stefan