What does `undecided' do for encoding text?

What does `undecided' do for encoding text?

[Eli Zaretskii]

It looks like `undecided' works the same as `raw-text' for encoding
text.  Is that true in general?  If not, what _does_ `undecided' do in
this case?

I think this should be reflected in the ELisp manual.

Re: What does `undecided' do for encoding text?

[Kenichi Handa]

In article <u4oz6tg7p.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> It looks like `undecided' works the same as `raw-text' for encoding
> text.  Is that true in general?

Yes.  More exactly, it is the same as `raw-text-unix'.

> I think this should be reflected in the ELisp manual.

I don't think so because it is just a fallback behavior, and
Elisp programmer should avoid specifying `undecided' on
encoding.

---
Kenichi Handa
handa@m17n.org

Re: What does `undecided' do for encoding text?

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Tue, 10 Feb 2009 16:01:10 +0900
> 
> In article <u4oz6tg7p.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It looks like `undecided' works the same as `raw-text' for encoding
> > text.  Is that true in general?
> 
> Yes.  More exactly, it is the same as `raw-text-unix'.

Thanks.

> > I think this should be reflected in the ELisp manual.
> 
> I don't think so because it is just a fallback behavior, and
> Elisp programmer should avoid specifying `undecided' on
> encoding.

This may have nothing to do with what the programmer did.  My use case
was in Rmail: if the original message had non-ASCII text encoded with
QP or B64, Rmail would (correctly) return `undecided' when it detects
the message encoding.  Suppose you then edit the message and add
non-ASCII characters as raw bytes, e.g. by base64-decode-region, and
type "C-c C-c".  rmail-cease-edit now needs to encode the text and put
it back into the mbox buffer, and the immediate choice it has for the
pertinent coding-system is to use buffer-file-coding-system of the
buffer where the message was edited.  But the value of
buffer-file-coding-system in that buffer is `undecided'...

If you say that using `undecided' in encoding is ``considered
harmful'', we should at least say that in the ELisp manual.  Although
in the use case I described `undecided' did exactly what's right, and
the only other correct choice is `raw-text'.

Re: What does `undecided' do for encoding text?

[Kenichi Handa]

In article <u63jisl2w.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > I don't think so because it is just a fallback behavior, and
> > Elisp programmer should avoid specifying `undecided' on
> > encoding.

> This may have nothing to do with what the programmer did.  My use case
> was in Rmail: if the original message had non-ASCII text encoded with
> QP or B64, Rmail would (correctly) return `undecided' when it detects
> the message encoding.  Suppose you then edit the message and add
> non-ASCII characters as raw bytes, e.g. by base64-decode-region, and
> type "C-c C-c".  rmail-cease-edit now needs to encode the text and put
> it back into the mbox buffer, and the immediate choice it has for the
> pertinent coding-system is to use buffer-file-coding-system of the
> buffer where the message was edited.  But the value of
> buffer-file-coding-system in that buffer is `undecided'...

The situation is the same as visiting an ASCII only file,
entering some non-ASCII characters, and saving it.  In that
case, we use selecte-safe-coding-system to decide the coding
system to encode the buffer contents.  As I wrote before,
rmail-cease-edit should do the same or the similar thing.

> If you say that using `undecided' in encoding is ``considered
> harmful'', we should at least say that in the ELisp manual.  Although
> in the use case I described `undecided' did exactly what's right, and
> the only other correct choice is `raw-text'.

For a buffer that contains raw bytes, `raw-text' is the only
correct choice.

---
Kenichi Handa
handa@m17n.org

Re: What does `undecided' do for encoding text?

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Tue, 10 Feb 2009 20:16:53 +0900
> 
> > This may have nothing to do with what the programmer did.  My use case
> > was in Rmail: if the original message had non-ASCII text encoded with
> > QP or B64, Rmail would (correctly) return `undecided' when it detects
> > the message encoding.  Suppose you then edit the message and add
> > non-ASCII characters as raw bytes, e.g. by base64-decode-region, and
> > type "C-c C-c".  rmail-cease-edit now needs to encode the text and put
> > it back into the mbox buffer, and the immediate choice it has for the
> > pertinent coding-system is to use buffer-file-coding-system of the
> > buffer where the message was edited.  But the value of
> > buffer-file-coding-system in that buffer is `undecided'...
> 
> The situation is the same as visiting an ASCII only file,
> entering some non-ASCII characters, and saving it.  In that
> case, we use selecte-safe-coding-system to decide the coding
> system to encode the buffer contents.

I think that's a different situation: base64-decode-region inserts raw
bytes, not non-ASCII characters.  So select-safe-coding-system may not
work correctly, because (AFAIK) it was not designed for that case.

Re: What does `undecided' do for encoding text?

[Stefan Monnier]

>> It looks like `undecided' works the same as `raw-text' for encoding
>> text.  Is that true in general?
> Yes.  More exactly, it is the same as `raw-text-unix'.

Which I prefer calling `binary'.

>> I think this should be reflected in the ELisp manual.
> I don't think so because it is just a fallback behavior, and
> Elisp programmer should avoid specifying `undecided' on
> encoding.

Then why do we even allow it?  We should signal an error.


        Stefan

Re: What does `undecided' do for encoding text?

[Kenichi Handa]

In article <uocxatswu.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: Kenichi Handa <handa@m17n.org>
> > CC: emacs-devel@gnu.org
> > Date: Tue, 10 Feb 2009 20:16:53 +0900
> > 
> > > This may have nothing to do with what the programmer did.  My use case
> > > was in Rmail: if the original message had non-ASCII text encoded with
> > > QP or B64, Rmail would (correctly) return `undecided' when it detects
> > > the message encoding.  Suppose you then edit the message and add
> > > non-ASCII characters as raw bytes, e.g. by base64-decode-region, and
> > > type "C-c C-c".  rmail-cease-edit now needs to encode the text and put
> > > it back into the mbox buffer, and the immediate choice it has for the
> > > pertinent coding-system is to use buffer-file-coding-system of the
> > > buffer where the message was edited.  But the value of
> > > buffer-file-coding-system in that buffer is `undecided'...
> > 
> > The situation is the same as visiting an ASCII only file,
> > entering some non-ASCII characters, and saving it.  In that
> > case, we use selecte-safe-coding-system to decide the coding
> > system to encode the buffer contents.

> I think that's a different situation: base64-decode-region inserts raw
> bytes, not non-ASCII characters.  So select-safe-coding-system may not
> work correctly, because (AFAIK) it was not designed for that case.

Have you tried it?  select-safe-coding-system should suggest
raw-text if the buffer contains raw bytes.

---
Kenichi Handa
handa@m17n.org

Re: What does `undecided' do for encoding text?

[Kenichi Handa]

In article <jwvwsbykkzu.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> It looks like `undecided' works the same as `raw-text' for encoding
>>> text.  Is that true in general?
> > Yes.  More exactly, it is the same as `raw-text-unix'.

> Which I prefer calling `binary'.

What do you mean by "calling"?

raw-text-unix and binary are different when
coding-system-change-text-conversion and
coding-system-change-eol-conversion are called.

>>> I think this should be reflected in the ELisp manual.
> > I don't think so because it is just a fallback behavior, and
> > Elisp programmer should avoid specifying `undecided' on
> > encoding.

> Then why do we even allow it?  We should signal an error.

It seems difficult to find a good point of signaling an
error.  At least, visiting an ascii only file, inputting
ascii only, and saving it should be a valid operation.  At
that time, buffer-file-coding-system is undecided-unix.

---
Kenichi Handa
handa@m17n.org

Re: What does `undecided' do for encoding text?

[Stefan Monnier]

>>>> It looks like `undecided' works the same as `raw-text' for encoding
>>>> text.  Is that true in general?
>> > Yes.  More exactly, it is the same as `raw-text-unix'.
>> Which I prefer calling `binary'.

> What do you mean by "calling"?

> raw-text-unix and binary are different when
> coding-system-change-text-conversion and
> coding-system-change-eol-conversion are called.

Good point, I tend to forget about this.  But in the context of "how is
`undecided' treated during write", this doesn't make any difference.

>>>> I think this should be reflected in the ELisp manual.
>> > I don't think so because it is just a fallback behavior, and
>> > Elisp programmer should avoid specifying `undecided' on
>> > encoding.

>> Then why do we even allow it?  We should signal an error.

> It seems difficult to find a good point of signaling an
> error.  At least, visiting an ascii only file, inputting
> ascii only, and saving it should be a valid operation.  At
> that time, buffer-file-coding-system is undecided-unix.

Then an error should only be signalled when we end up trying to encode
a non-ASCII char.


        Stefan