bug#34909: 26.1; Error refreshing packages under language environment

[Eli Zaretskii <eliz@gnu.org>]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: choroba@matfyz.cz,  34909@debbugs.gnu.org
> Date: Tue, 19 Mar 2019 15:41:27 -0400
> 
> > (I also don't understand why you say the buffer must be unibyte: if
> > the coding-systems for each operation are set correctly, there should
> > be no difference at all, not nowadays.  IME, using unibyte buffers is
> > just the easy way out when one doesn't want to mess with the
> > coding-systems stuff.)
> 
> Of course, when dealing with data we only transfer from network to file,
> using unibyte (or more specifically: without decoding nor encoding) is
> simply more efficient, but it's also safer: Encoding using utf-8 is only
> safe if the decoding was itself done using the same coding system and
> I don't see where we do that, so it's not clear that it's always decoded
> with utf-8.

Did you also look in url-*.el stuff, which package.el invokes?

My motivation to use UTF-8 instead of making the buffer unibyte was
that select-safe-coding-system popped up a buffer which clearly showed
characters, not raw bytes, and it offered to select UTF-8, not
raw-text.  This can mean only one thing: that text in the buffer is
already correctly represented, and the problem was with an attempt to
encode it using a coding-system which cannot handle some of the text
(specifically, Chinese characters and a few Latin characters not
supported by ISO 8859-2).

> > Feel free to work on package--with-response-buffer in this direction.
> > Debugging this was a small nightmare, due to the use of macros and
> > async retrieval with callbacks, so I felt lucky when I finally found
> > the problematic code and understood why it worked in the English
> > language environment.
> 
> "nightmare" is very appropriate, indeed.

I will just add that the wider becomes our use of sophisticated Lisp
techniques in core packages, the harder it will be to debug our own
code.  I really wish someone would start some serious work on
expanding our debugging facilities and making them adequate for
debugging the kind of code we see in package.el.  'Nough said.