package.el encoding problem

package.el encoding problem

[Yuri D'Elia]

package.el seems to have a fit with this melpa package:

https://melpa.org/packages/web-mode-20190522.610.el

When attempting to install, it fails to determine the encoding of the
file and prompts for one. However when downloading it by hand it does
seem to be correctly coded as utf-8.

Re: package.el encoding problem

[Noam Postavsky]

On Thu, 23 May 2019 at 11:24, Yuri D'Elia <wavexx@thregr.org> wrote:
>
> package.el seems to have a fit with this melpa package:
>
> https://melpa.org/packages/web-mode-20190522.610.el
>
> When attempting to install, it fails to determine the encoding of the
> file and prompts for one. However when downloading it by hand it does
> seem to be correctly coded as utf-8.

Is this a package.el of the latest master (I think Stefan recently
pushed some changes that affect decoding) or an earlier one?

Re: package.el encoding problem

[Yuri D'Elia]

On Thu, May 23 2019, Noam Postavsky wrote:
> On Thu, 23 May 2019 at 11:24, Yuri D'Elia <wavexx@thregr.org> wrote:
>>
>> package.el seems to have a fit with this melpa package:
>>
>> https://melpa.org/packages/web-mode-20190522.610.el
>>
>> When attempting to install, it fails to determine the encoding of the
>> file and prompts for one. However when downloading it by hand it does
>> seem to be correctly coded as utf-8.
>
> Is this a package.el of the latest master (I think Stefan recently
> pushed some changes that affect decoding) or an earlier one?

emacs master from ~30 minutes ago (5424436452bc0b3d8a62a8398f92d0c2db81e22b)

Re: package.el encoding problem

[Stefan Monnier]

> package.el seems to have a fit with this melpa package:
> https://melpa.org/packages/web-mode-20190522.610.el
> When attempting to install, it fails to determine the encoding of the
> file and prompts for one.

Can you enable "Options => Enter Debugger on Quit", then reproduce the
problem then hit C-g when you get the prompt?

[ Or do `M-: (debug)` when you get the prompt.  ]

You might also do this within a formal bug-report, so we get a tracking
number for it.


        Stefan

Re: package.el encoding problem

[Stefan Monnier]

> Can you enable "Options => Enter Debugger on Quit", then reproduce the
> problem then hit C-g when you get the prompt?
> [ Or do `M-: (debug)` when you get the prompt.  ]

Don't bother I managed to reproduce it after all.  It should fixed, thanks,


        Stefan

Re: package.el encoding problem

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 24 May 2019 12:36:01 -0400
> 
> > Can you enable "Options => Enter Debugger on Quit", then reproduce the
> > problem then hit C-g when you get the prompt?
> > [ Or do `M-: (debug)` when you get the prompt.  ]
> 
> Don't bother I managed to reproduce it after all.  It should fixed, thanks,

Thanks, but I don't think I understand the fix.  Could you explain why
making the temporary buffer unibyte solves the problem?  I thought the
need for unibyte buffers was removed long ago, at least in the vast
majority of situations.

Re: package.el encoding problem

[Stefan Monnier]

>> > Can you enable "Options => Enter Debugger on Quit", then reproduce the
>> > problem then hit C-g when you get the prompt?
>> > [ Or do `M-: (debug)` when you get the prompt.  ]
>> Don't bother I managed to reproduce it after all.  It should fixed, thanks,
> Thanks, but I don't think I understand the fix.

As you can see in the fix's assertion, the data we receive is
a unibyte string and we need to save it into a file.

We used to put it into a multibyte buffer, which then causes the save
the be all confused because the bytes 128-255 it contains aren't part of
any coding-system.  We did try to circumvent this problem by specifying
`no-conversion` coding system, but I think the way we did this wasn't
quite right.

Rather than try to fix the circumvention, I decided to "do it right" and
use a unibyte buffer so the question doesn't show up.

> I thought the need for unibyte buffers was removed long ago, at least
> in the vast majority of situations.

It is definitely *possible* to use multibyte buffers even in cases where
we only manipulate bytes, but it is undesirable.


        Stefan

Re: package.el encoding problem

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sat, 25 May 2019 08:15:52 -0400
> 
> >> > Can you enable "Options => Enter Debugger on Quit", then reproduce the
> >> > problem then hit C-g when you get the prompt?
> >> > [ Or do `M-: (debug)` when you get the prompt.  ]
> >> Don't bother I managed to reproduce it after all.  It should fixed, thanks,
> > Thanks, but I don't think I understand the fix.
> 
> As you can see in the fix's assertion, the data we receive is
> a unibyte string and we need to save it into a file.

Yes, which is why I said I didn't understand the fix.  Multibyte
buffers can handle raw bytes without any problem.  Or at least I
thought they did.

> We used to put it into a multibyte buffer, which then causes the save
> the be all confused because the bytes 128-255 it contains aren't part of
> any coding-system.

I don't see how that matters.  Raw bytes should be converted back to
their original unibyte form when saving, no matter what coding-system
is used.

Could you perhaps show a recipe for the problem?  I'd like to look
into what happens there.

> It is definitely *possible* to use multibyte buffers even in cases where
> we only manipulate bytes, but it is undesirable.

I'm probably missing something, because I don't see would that be
undesirable.  Hopefully, a reproducible recipe will show me the light.

Re: package.el encoding problem

[Stefan Monnier]

>> We used to put it into a multibyte buffer, which then causes the save
>> the be all confused because the bytes 128-255 it contains aren't part of
>> any coding-system.
>
> I don't see how that matters.  Raw bytes should be converted back to
> their original unibyte form when saving, no matter what coding-system
> is used.

The bytes were saved correctly.  But before that happened, the user was
prompted to choose a coding-system.

>> It is definitely *possible* to use multibyte buffers even in cases where
>> we only manipulate bytes, but it is undesirable.
> I'm probably missing something, because I don't see would that be
> undesirable.

It doesn't do anything else than introduce problems (e.g. having to
decide how to encode chars even though there are no chars to decode)
and inefficiencies.

I'm rather curious what you think would be the benefits from using
a multibyte buffer here.


        Stefan

Re: package.el encoding problem

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sat, 25 May 2019 10:55:36 -0400
> 
> >> We used to put it into a multibyte buffer, which then causes the save
> >> the be all confused because the bytes 128-255 it contains aren't part of
> >> any coding-system.
> >
> > I don't see how that matters.  Raw bytes should be converted back to
> > their original unibyte form when saving, no matter what coding-system
> > is used.
> 
> The bytes were saved correctly.  But before that happened, the user was
> prompted to choose a coding-system.

Is that the only problem?  IOW, if we prevent the prompt, will
package.el work correctly in this scenario?

You previously said that in the past we attempted to bind
coding-system-for-write, which in general is the easiest way of
preventing the prompt.  Didn't it work?

> >> It is definitely *possible* to use multibyte buffers even in cases where
> >> we only manipulate bytes, but it is undesirable.
> > I'm probably missing something, because I don't see would that be
> > undesirable.
> 
> It doesn't do anything else than introduce problems (e.g. having to
> decide how to encode chars even though there are no chars to decode)
> and inefficiencies.
> 
> I'm rather curious what you think would be the benefits from using
> a multibyte buffer here.

One obvious benefit is that you won't need to set the buffer to be
unibyte.  People tend to regard this as some kind of black magic,
which creates myths, like the (wrong) idea that unibyte text cannot be
processed correctly in a multibyte buffer.  I'd rather we avoided
substantiating such myths.  Also, we should in theory be able to
eliminate unibyte buffers, at least in Lisp application code.

Re: package.el encoding problem

[Stefan Monnier]

> You previously said that in the past we attempted to bind
> coding-system-for-write, which in general is the easiest way of
> preventing the prompt.  Didn't it work?

For some reason this function does something else:

    (defun package--write-file-no-coding (file-name)
      (let ((buffer-file-coding-system 'no-conversion))
        (write-region (point-min) (point-max) file-name nil 'silent)))

AFAIK using coding-system-for-write would have solve the problem as well.

> One obvious benefit is that you won't need to set the buffer to be
> unibyte.

(set-buffer-multibyte nil) sets up the buffer to receive bytes.
Since we're putting bytes into the buffer, it's The Right Thing to do.

> People tend to regard this as some kind of black magic, which creates
> myths, like the (wrong) idea that unibyte text cannot be processed
> correctly in a multibyte buffer.

I think they're right: it's hard to get it right.  Partly because it
encourages confusion between bytes and chars (and confusion between
sequences of bytes and sequences of chars).

Don't get me wrong: it's important that it be possible to do it, because
that's sometimes necessary.  But when the code only manipulates bytes,
using any multibyte objects along the way is asking for trouble.

> I'd rather we avoided substantiating such myths.

And I'd rather we clarify that chars aren't bytes and vice versa.
[ I also wish we could through away the "unibyte" and "multibyte"
  vocabulary which again encourages such confusion.  ]

> Also, we should in theory be able to eliminate unibyte buffers, at
> least in Lisp application code.

I think this will help create more confusion.


        Stefan

Re: package.el encoding problem

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 25 May 2019 11:26:21 -0400
> 
> > You previously said that in the past we attempted to bind
> > coding-system-for-write, which in general is the easiest way of
> > preventing the prompt.  Didn't it work?
> 
> For some reason this function does something else:
> 
>     (defun package--write-file-no-coding (file-name)
>       (let ((buffer-file-coding-system 'no-conversion))
>         (write-region (point-min) (point-max) file-name nil 'silent)))
> 
> AFAIK using coding-system-for-write would have solve the problem as well.

Yes.  In fact, binding buffer-file-coding-system is not useful at all.

So why not bind coding-system-for-write here?

> > One obvious benefit is that you won't need to set the buffer to be
> > unibyte.
> 
> (set-buffer-multibyte nil) sets up the buffer to receive bytes.
> Since we're putting bytes into the buffer, it's The Right Thing to do.

We disagree here.  We can put bytes into a multibyte buffer, they have
a special representation there which tells they are raw bytes, so
there's nothing wrong with doing that.

> > People tend to regard this as some kind of black magic, which creates
> > myths, like the (wrong) idea that unibyte text cannot be processed
> > correctly in a multibyte buffer.
> 
> I think they're right: it's hard to get it right.

Using a multibyte buffer removes one subtlety from what we should
educate people to do in this case, so it's easier to get that right.

> Don't get me wrong: it's important that it be possible to do it, because
> that's sometimes necessary.  But when the code only manipulates bytes,
> using any multibyte objects along the way is asking for trouble.

Sorry, I see no trouble here.  the only trouble is the binding of
buffer-file-coding-system instead of coding-system-for-write.