^M in the info files

^M in the info files

[Lennart Borgman (gmail)]

I see a lot of ^M in the info files on w32, CVS from today. For example 
in these files

   Emacs FAQ
   VIPER

Re: ^M in the info files

[dhruva]

Same here, I kept quiet as I am using GIT and thought I had messed
around with something!

On Wed, Jun 11, 2008 at 6:13 AM, Lennart Borgman (gmail)
<lennart.borgman@gmail.com> wrote:
> I see a lot of ^M in the info files on w32, CVS from today. For example in
> these files

-dhruva

-- 
Contents reflect my personal views only!

Re: ^M in the info files

[Juanma Barranquero]

On Wed, Jun 11, 2008 at 06:42, dhruva <dhruvakm@gmail.com> wrote:

> Same here, I kept quiet as I am using GIT and thought I had messed
> around with something!
>
> On Wed, Jun 11, 2008 at 6:13 AM, Lennart Borgman (gmail)
> <lennart.borgman@gmail.com> wrote:
>> I see a lot of ^M in the info files on w32, CVS from today. For example in
>> these files

Apparently related to these changes:

2008-06-05  Kenichi Handa  <handa@m17n.org>

	* coding.c (detect_coding): Fix previous change.
	(detect_coding_system): Likewise.

2008-06-04  Kenichi Handa  <handa@m17n.org>

	* coding.c (detect_coding): Fix handling of coding->head_ascii.
	Be sure to call setup_coding_system when we find a proper coding system.
	(detect_coding_system): Fix handling of coding->head_ascii.

Removing them fixes the problem for me.

  Juanma

Re: ^M in the info files

[Juanma Barranquero]

Has this one been forgotten?

On Wed, Jun 11, 2008 at 17:56, Juanma Barranquero <lekktu@gmail.com> wrote:
> On Wed, Jun 11, 2008 at 06:42, dhruva <dhruvakm@gmail.com> wrote:
>
>> Same here, I kept quiet as I am using GIT and thought I had messed
>> around with something!
>>
>> On Wed, Jun 11, 2008 at 6:13 AM, Lennart Borgman (gmail)
>> <lennart.borgman@gmail.com> wrote:
>>> I see a lot of ^M in the info files on w32, CVS from today. For example in
>>> these files
>
> Apparently related to these changes:
>
> 2008-06-05  Kenichi Handa  <handa@m17n.org>
>
>        * coding.c (detect_coding): Fix previous change.
>        (detect_coding_system): Likewise.
>
> 2008-06-04  Kenichi Handa  <handa@m17n.org>
>
>        * coding.c (detect_coding): Fix handling of coding->head_ascii.
>        Be sure to call setup_coding_system when we find a proper coding system.
>        (detect_coding_system): Fix handling of coding->head_ascii.
>
> Removing them fixes the problem for me.
>
>  Juanma

Re: ^M in the info files

[Kenichi Handa]

In article <f7ccd24b0807081851j30dd8a61p4f1926dd740f033b@mail.gmail.com>, "Juanma Barranquero" <lekktu@gmail.com> writes:

> Has this one been forgotten?

No, but I haven't had a time to work on it.  As I've just
committed a change about font selection, I started to work
on it.  I've just installed the latest Emacs on Windows with
cygwin, and done this:

> The sequence is this:

>  - I have a ChangeLog, apparently correct, up-to-date with the repository.
>  - I modify it, typically by adding a new ChangeLog entry.
>  - I commit in from inside Emacs, with vc-next-action followed by log-edit-done.
>  - I see the diff in emacs-diffs and notice that an empty line has
> been deleted. This seems to happen more to empty lines that separate
> paragraphs from date/author lines (as opposed to empty space between
> paragraphs), but I have no hard data, just a feeling.
>  - I edit the ChangeLog to see what's happened. All the lines end in ^M.
>  - I remove the ^M (with replace-string <ENTER> ^M^J <ENTER> ^J) [I
> don't write the ChangeLog, it's just to make it easier spot problems.]
>  - At that point (after removing the ^Ms) the line with the problem
> has this aspect:

But, I can't reproduce the bug.  The diff of the step 4
above shows only the entry I added.  What was the coding
system of your ChangeLog file when you first visitted it?

---
Kenichi Handa
handa@ni.aist.go.jp


> On Wed, Jun 11, 2008 at 17:56, Juanma Barranquero <lekktu@gmail.com> wrote:
> > On Wed, Jun 11, 2008 at 06:42, dhruva <dhruvakm@gmail.com> wrote:
> >
>>> Same here, I kept quiet as I am using GIT and thought I had messed
>>> around with something!
>>> 
>>> On Wed, Jun 11, 2008 at 6:13 AM, Lennart Borgman (gmail)
>>> <lennart.borgman@gmail.com> wrote:
>>>> I see a lot of ^M in the info files on w32, CVS from today. For example in
>>>> these files
> >
> > Apparently related to these changes:
> >
> > 2008-06-05  Kenichi Handa  <handa@m17n.org>
> >
> >        * coding.c (detect_coding): Fix previous change.
> >        (detect_coding_system): Likewise.
> >
> > 2008-06-04  Kenichi Handa  <handa@m17n.org>
> >
> >        * coding.c (detect_coding): Fix handling of coding->head_ascii.
> >        Be sure to call setup_coding_system when we find a proper coding system.
> >        (detect_coding_system): Fix handling of coding->head_ascii.
> >
> > Removing them fixes the problem for me.
> >
> >  Juanma

Re: ^M in the info files

[Juanma Barranquero]

On Wed, Jul 9, 2008 at 04:44, Kenichi Handa <handa@m17n.org> wrote:

> No, but I haven't had a time to work on it.

Sorry if it seemed a request; I was just making sure it didn't went lost.

>> The sequence is this:
>
>>  - I have a ChangeLog, apparently correct, up-to-date with the repository.
>>  - I modify it, typically by adding a new ChangeLog entry.
>>  - I commit in from inside Emacs, with vc-next-action followed by log-edit-done.
>>  - I see the diff in emacs-diffs and notice that an empty line has
>> been deleted. This seems to happen more to empty lines that separate
>> paragraphs from date/author lines (as opposed to empty space between
>> paragraphs), but I have no hard data, just a feeling.
>>  - I edit the ChangeLog to see what's happened. All the lines end in ^M.
>>  - I remove the ^M (with replace-string <ENTER> ^M^J <ENTER> ^J) [I
>> don't write the ChangeLog, it's just to make it easier spot problems.]
>>  - At that point (after removing the ^Ms) the line with the problem
>> has this aspect:

I think you're mixing two different bugs. The one I referred to in the
"has [...] been forgotten" message is about ^M in info files, which
happens right now in the Windows port (there has been at least three
reporters, including me).

> But, I can't reproduce the bug.  The diff of the step 4
> above shows only the entry I added.

What you quote above is from a problem with ChangeLogs that only I
see, apparently, so it's no wonder you cannot reproduce it. *I* cannot
reproduce it at will, alas...

> What was the coding
> system of your ChangeLog file when you first visitted it?

utf-8-dos, AFAICS.

  Juanma

Re: ^M in the info files

[Kenichi Handa]

In article <f7ccd24b0807081956o1c443d54k64cafc104d850aba@mail.gmail.com>, "Juanma Barranquero" <lekktu@gmail.com> writes:

> I think you're mixing two different bugs. The one I referred to in the
> "has [...] been forgotten" message is about ^M in info files, which
> happens right now in the Windows port (there has been at least three
> reporters, including me).

> > But, I can't reproduce the bug.  The diff of the step 4
> > above shows only the entry I added.

> What you quote above is from a problem with ChangeLogs that only I
> see, apparently, so it's no wonder you cannot reproduce it. *I* cannot
> reproduce it at will, alas...

Oops, sorry.  But, I can't see the ^M problem in info files.
Actually, on Windows, when I type C-h i, I get this error:

  Can't find the Info directory node

But, when I type C-u C-h i ~/emacs/info/efag RET (I built
Emacs under ~/emacs),  I see no '^M's.

What is the coding system of emacs/info/efag when you visit
that file directly?

> > What was the coding
> > system of your ChangeLog file when you first visitted it?

> utf-8-dos, AFAICS.

Hmmm, I tried with a ChangeLog of that encoding, but still
can't reproduce the bug.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: ^M in the info files

[Jason Rumney]

Kenichi Handa wrote:
> In article <f7ccd24b0807081956o1c443d54k64cafc104d850aba@mail.gmail.com>, "Juanma Barranquero" <lekktu@gmail.com> writes:

> Oops, sorry.  But, I can't see the ^M problem in info files.
> Actually, on Windows, when I type C-h i, I get this error:
> 
>   Can't find the Info directory node
> 
> But, when I type C-u C-h i ~/emacs/info/efag RET (I built
> Emacs under ~/emacs),  I see no '^M's.

Perhaps the version of makeinfo you have is not generating files with
DOS line ends. Mine is, and even opening info/efaq with C-x C-f shows
the ^M characters, though I don't see any inconsistencies that should
cause line-end detection to fail and Emacs 22.2 opens it correctly with
a DOS coding system.

Re: ^M in the info files

[Juanma Barranquero]

On Wed, Jul 9, 2008 at 06:33, Kenichi Handa <handa@m17n.org> wrote:

> What is the coding system of emacs/info/efag when you visit
> that file directly?

Both when I visit it directly and when I access it through Info,
buffer-file-coding-system is `no-conversion'.

> Hmmm, I tried with a ChangeLog of that encoding, but still
> can't reproduce the bug.

As I said, neither can I reliably.

  Juanma

Re: ^M in the info files

[Kenichi Handa]

In article <487481B2.3090302@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> Kenichi Handa wrote:
> > In article <f7ccd24b0807081956o1c443d54k64cafc104d850aba@mail.gmail.com>, "Juanma Barranquero" <lekktu@gmail.com> writes:

> > Oops, sorry.  But, I can't see the ^M problem in info files.
> > Actually, on Windows, when I type C-h i, I get this error:
> > 
> >   Can't find the Info directory node
> > 
> > But, when I type C-u C-h i ~/emacs/info/efag RET (I built
> > Emacs under ~/emacs),  I see no '^M's.

> Perhaps the version of makeinfo you have is not generating files with
> DOS line ends. Mine is,

It seems so.

> and even opening info/efaq with C-x C-f shows
> the ^M characters, though I don't see any inconsistencies that should
> cause line-end detection to fail and Emacs 22.2 opens it correctly with
> a DOS coding system.

I've just found that the file "efaq" contains null-bytes
('\0') after "Concept Index\n********\n\n".  "viper" also
contains null-bytes.  So, Emacs detects that those files are
binary.  Hmmm, what should we do?  Make a new variable
inhibit-null-byte-detection (analogous to
inhibit-iso-escape-detection)?  Or, make the code of
null-byte detection checks the percentage of null-byte, and
conclude that the file is binary only when the percentage is
higher than some threshold?

---
Kenichi Handa
handa@ni.aist.go.jp

"no-conversion" coding system (was: ^M in the info files)

[Stefan Monnier]

>> What is the coding system of emacs/info/efag when you visit
>> that file directly?

> Both when I visit it directly and when I access it through Info,
> buffer-file-coding-system is `no-conversion'.

BTW, could we get rid of `no-conversion' (or at least make it obsolete
and deprecated)?

I mean, this is a misnomer which gives the illusion that there is such
a thing as "no conversion", even though in reality there always is some
conversion going on.  Better use the name `binary', I think.


        Stefan

Re: ^M in the info files

[Stefan Monnier]

> I've just found that the file "efaq" contains null-bytes
> ('\0') after "Concept Index\n********\n\n".  "viper" also
> contains null-bytes.

Why is that?  Isn't it an error in those files?

> So, Emacs detects that those files are binary.

Does info.el need to use Emacs's auto-detection machinery?  I mean don't
Info files come with their own scheme to specify the encoding used?


        Stefan

Re: ^M in the info files

[James Cloos]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've just found that the file "efaq" contains null-bytes
>> ('\0') after "Concept Index\n********\n\n".  "viper" also
>> contains null-bytes.

Stefan> Why is that?  Isn't it an error in those files?

There are *many* such examples in my info directory.  230 out of 856
files in my /usr/share/info have them.  It looks like all of them
have one or more instances of the line (in cat --show-all syntax):

^@^H[index^@^H]$

In the case of faq.texi it is from the code:

@node Concept index,  , Mail and news, Top
@unnumbered Concept Index
@printindex cp

@contents

I have texinfo-4.12 installed, but some of the relevant files date back
as far as 2004/August, so it isn't just a recent thing.

From my most recent compile of emacs, it affects:

emacs-mime.info.bz2
gnus-5.info.bz2
message.info.bz2
pgg.info.bz2
sasl.info.bz2
sieve.info.bz2

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: ^M in the info files

[Juri Linkov]

> There are *many* such examples in my info directory.  230 out of 856
> files in my /usr/share/info have them.  It looks like all of them
> have one or more instances of the line (in cat --show-all syntax):
>
> ^@^H[index^@^H]$
>
> In the case of faq.texi it is from the code:
>
> @node Concept index,  , Mail and news, Top
> @unnumbered Concept Index
> @printindex cp
>
> @contents
>
> I have texinfo-4.12 installed, but some of the relevant files date back
> as far as 2004/August, so it isn't just a recent thing.

This is an old feature, so I wonder why it starts causing problems
just now.

It is also used for embedded images in Info files:

  ^@^H[image src="BINARYFILE" text="TXTFILE" alt="ALTTEXT ... ^@^H]

so with a lot of images the percentage of null-bytes may be too high.

The Info reader correctly treats the coding cookie, so one solution
is to write it to all Info files, but this does help with existing
files.

But I still doesn't understand the problem with ^@^H in Info files.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: "no-conversion" coding system (was: ^M in the info files)

[Kenichi Handa]

In article <jwvzlorqfgj.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> What is the coding system of emacs/info/efag when you visit
>>> that file directly?

> > Both when I visit it directly and when I access it through Info,
> > buffer-file-coding-system is `no-conversion'.

> BTW, could we get rid of `no-conversion' (or at least make it obsolete
> and deprecated)?

> I mean, this is a misnomer which gives the illusion that there is such
> a thing as "no conversion", even though in reality there always is some
> conversion going on.  Better use the name `binary', I think.

When we do C-x C-m c no-conversion RET C-x C-f FILE RET,
"no-conversion" really means "no-conversion" because
FILE is read into a unibyte buffer without any conversion.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: "no-conversion" coding system (was: ^M in the info files)

[Richard M Stallman]

    I mean, this is a misnomer which gives the illusion that there is such
    a thing as "no conversion", even though in reality there always is some
    conversion going on.

`no-conversion' used to really mean no conversion.
Why has that changed?

Re: "no-conversion" coding system (was: ^M in the info files)

[Kenichi Handa]

In article <E1KGjXh-0000U4-Jl@fencepost.gnu.org>, Richard M Stallman <rms@gnu.org> writes:

>     I mean, this is a misnomer which gives the illusion that there is such
>     a thing as "no conversion", even though in reality there always is some
>     conversion going on.

> `no-conversion' used to really mean no conversion.
> Why has that changed?

Nothing is changed.  When you insert a file in a multibyte
buffer with no-conversion, each eight-bit code is converted
to a special 2-byte form to represent eight-bit character.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: "no-conversion" coding system

[Stefan Monnier]

>> I mean, this is a misnomer which gives the illusion that there is such
>> a thing as "no conversion", even though in reality there always is some
>> conversion going on.

>> `no-conversion' used to really mean no conversion.
>> Why has that changed?

Technically, that may be true.  But take a utf-8 text and open it with
"no-conversion" and it won't look like the same text, so for some
interpretation of "conversion", it has been converted.
I.e. it's a name that leads to confusion.  Its other name "binary" is
a lot more unequivocal.


        Stefan

Re: ^M in the info files

[Kenichi Handa]

In article <jwvtzezovh5.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Does info.el need to use Emacs's auto-detection machinery?  I mean don't
> Info files come with their own scheme to specify the encoding used?

FYI.  Only gnus.texi and emacs-mime.texi have
@documentencoding directive along with coding: tag.  As a
result (or not, I'm not sure), the resulting info files
"gnus" and "emacs-mime" contain coding: tag.

And, for instance, faq.texi has @today{} directive, and it
seems that makeinfo generates a date string according to the
current locale (and thus results in non-ASCII characters).

---
Kenichi Handa
handa@ni.aist.go.jp

Re: ^M in the info files

[Stefan Monnier]

>> Does info.el need to use Emacs's auto-detection machinery?  I mean don't
>> Info files come with their own scheme to specify the encoding used?

> FYI.  Only gnus.texi and emacs-mime.texi have
> @documentencoding directive along with coding: tag.  As a
> result (or not, I'm not sure), the resulting info files
> "gnus" and "emacs-mime" contain coding: tag.

But the others are pure ASCII aren't they (isn't that what it means for
an Info file not to have a "coding:" tag)?

> And, for instance, faq.texi has @today{} directive, and it
> seems that makeinfo generates a date string according to the
> current locale (and thus results in non-ASCII characters).

Isn't that a bug in makeinfo?


        Stefan

Re: ^M in the info files

[Juri Linkov]

>>> Does info.el need to use Emacs's auto-detection machinery?  I mean don't
>>> Info files come with their own scheme to specify the encoding used?
>
>> FYI.  Only gnus.texi and emacs-mime.texi have
>> @documentencoding directive along with coding: tag.  As a
>> result (or not, I'm not sure), the resulting info files
>> "gnus" and "emacs-mime" contain coding: tag.
>
> But the others are pure ASCII aren't they (isn't that what it means for
> an Info file not to have a "coding:" tag)?

If an Info file has no "coding:" tag, then the Info reader uses
Emacs's auto-detection machinery.

But maybe in this case we should force some safe coding like `undecided'
(as it seems to use now for pure ASCII Info files without null-bytes)?

>> And, for instance, faq.texi has @today{} directive, and it
>> seems that makeinfo generates a date string according to the
>> current locale (and thus results in non-ASCII characters).
>
> Isn't that a bug in makeinfo?

IIUC, this is an intentional feature.  But we could run it with e.g.
`LANG=C makeinfo' in Makefiles.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: ^M in the info files

[Stefan Monnier]

>>>> Does info.el need to use Emacs's auto-detection machinery?  I mean don't
>>>> Info files come with their own scheme to specify the encoding used?
>> 
>>> FYI.  Only gnus.texi and emacs-mime.texi have
>>> @documentencoding directive along with coding: tag.  As a
>>> result (or not, I'm not sure), the resulting info files
>>> "gnus" and "emacs-mime" contain coding: tag.
>> 
>> But the others are pure ASCII aren't they (isn't that what it means for
>> an Info file not to have a "coding:" tag)?

> If an Info file has no "coding:" tag, then the Info reader uses
> Emacs's auto-detection machinery.

I'm not talking about what Emacs does but about what the Info format
"specifies".  IIUC the Info format is always ASCII unless explicitly
specified by a conding: tag.  Of course, you can have an Info file
without a coding: tag that uses non-ASCII chars in some encoding, but
IIUC this has never been considered as valid (from TeXinfo's point
of view).

> But maybe in this case we should force some safe coding like `undecided'
> (as it seems to use now for pure ASCII Info files without null-bytes)?

No, I think it should use `us-ascii' instead.

>>> And, for instance, faq.texi has @today{} directive, and it
>>> seems that makeinfo generates a date string according to the
>>> current locale (and thus results in non-ASCII characters).
>> 
>> Isn't that a bug in makeinfo?

> IIUC, this is an intentional feature.  But we could run it with e.g.
> `LANG=C makeinfo' in Makefiles.

What happens if your TeXinfo file specifies a latin-1 encoding and your
date is output in utf-8 because of your locale, then?


        Stefan

Re: ^M in the info files

[Juri Linkov]

> I'm not talking about what Emacs does but about what the Info format
> "specifies".  IIUC the Info format is always ASCII unless explicitly
> specified by a conding: tag.  Of course, you can have an Info file
> without a coding: tag that uses non-ASCII chars in some encoding, but
> IIUC this has never been considered as valid (from TeXinfo's point
> of view).

Yes, it seems the default coding for Info files is supposed to be
US-ASCII unless overridden by @documentencoding.

>>>> And, for instance, faq.texi has @today{} directive, and it
>>>> seems that makeinfo generates a date string according to the
>>>> current locale (and thus results in non-ASCII characters).
>>>
>>> Isn't that a bug in makeinfo?
>
>> IIUC, this is an intentional feature.  But we could run it with e.g.
>> `LANG=C makeinfo' in Makefiles.
>
> What happens if your TeXinfo file specifies a latin-1 encoding and your
> date is output in utf-8 because of your locale, then?

Then the result of @today is displayed as garbage.

But we can adjust Makefile to use the correct LANG for every Info manual
that specifies a non-default @documentencoding.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: ^M in the info files

[Stefan Monnier]

>> What happens if your TeXinfo file specifies a latin-1 encoding and your
>> date is output in utf-8 because of your locale, then?

> Then the result of @today is displayed as garbage.

Again, my question is not how it's displayed but what is the content of
the resulting Info file.  If it's a mix of the two encodings, then it's
a bug in makeinfo (call it misfeature if you want) and I see no need to
try and support it in Emacs.

> But we can adjust Makefile to use the correct LANG for every Info manual
> that specifies a non-default @documentencoding.

That might be a good workaround,


        Stefan

Re: ^M in the info files

[Kenichi Handa]

In article <87k5fu94vx.fsf@jurta.org>, Juri Linkov <juri@jurta.org> writes:

> > I have texinfo-4.12 installed, but some of the relevant files date back
> > as far as 2004/August, so it isn't just a recent thing.

> This is an old feature, so I wonder why it starts causing problems
> just now.

I believe that it's because of the introduction of null-byte
detection code, but it was done in this April.  What I don't
understand is that....

Juanma wrote:

>>> On Wed, Jun 11, 2008 at 6:13 AM, Lennart Borgman (gmail)
>>> <lennart.borgman@gmail.com> wrote:
>>>> I see a lot of ^M in the info files on w32, CVS from today. For example in
>>>> these files
> >
> > Apparently related to these changes:
> >
> > 2008-06-05  Kenichi Handa  <handa@m17n.org>
> >
> >        * coding.c (detect_coding): Fix previous change.
> >        (detect_coding_system): Likewise.
> >
> > 2008-06-04  Kenichi Handa  <handa@m17n.org>
> >
> >        * coding.c (detect_coding): Fix handling of coding->head_ascii.
> >        Be sure to call setup_coding_system when we find a proper coding system.
> >        (detect_coding_system): Fix handling of coding->head_ascii.
> >
> > Removing them fixes the problem for me.
> >
> >  Juanma

I've just checked out the Emacs of 2008-06-03 (it doesn't
have the above change), and still info/efag is detected as
no-conversion.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: "no-conversion" coding system

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 09 Jul 2008 21:12:41 -0400
> Cc: lekktu@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> Technically, that may be true.  But take a utf-8 text and open it with
> "no-conversion" and it won't look like the same text, so for some
> interpretation of "conversion", it has been converted.

no-conversion doesn't mean that the text will _look_ the same, it
means the byte stream will be the same.

> Its other name "binary" is a lot more unequivocal.

Only if you are a programmer who knows that binary files are usually
read with no conversions.  Otherwise, the name "binary" doesn't have
any useful mnemonic meaning in the context of transforming one text
encoding into another.

Re: "no-conversion" coding system

[Stefan Monnier]

>> Its other name "binary" is a lot more unequivocal.
> Only if you are a programmer who knows that binary files are usually
> read with no conversions.  Otherwise, the name "binary" doesn't have

Since the `no-conversion' method depends on the underlying byte-encoding
used in a file, it is inherently something for programmers,
bit-twiddlers, ...


        Stefan

Re: "no-conversion" coding system

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: Stefan Monnier <monnier@iro.umontreal.ca>
 > > Date: Wed, 09 Jul 2008 21:12:41 -0400
 > > Cc: lekktu@gmail.com, rms@gnu.org, emacs-devel@gnu.org
 > > 
 > > Technically, that may be true.  But take a utf-8 text and open it with
 > > "no-conversion" and it won't look like the same text, so for some
 > > interpretation of "conversion", it has been converted.

Irrelevant.  You can achieve the same kind of confusion with `emacs
-font'. :-)

 > no-conversion doesn't mean that the text will _look_ the same, it
 > means the byte stream will be the same.

That's true as a definition, and incorrect as a matter of fact in
multibyte buffers.  Unless Emacs enforces "no-conversion is unibyte",
"no-conversion" should die, and split its estate between "raw-text"
(where newline conventions are meaningful) and "binary" (or
"raw-bytes") for non-text files.

 > > Its other name "binary" is a lot more unequivocal.
 > 
 > Only if you are a programmer who knows that binary files are usually
 > read with no conversions.  Otherwise, the name "binary" doesn't have
 > any useful mnemonic meaning in the context of transforming one text
 > encoding into another.

Binary isn't a text encoding, and therefore it's entirely appropriate
that it lack a mnemonic meaning in the context of text encoding.<wink>

In any case, only Emacs maintainers should ever need to care about
the representational transformations performed by coding systems, and
they all do know what binary means.  Everybody else just needs to know
the magic spell that turns mojibake into language; even if they did
know about representations, they can't do anything about it.  (I guess
that's not strictly so in Emacs, but it should be. ;-)

Re: ^M in the info files

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Thu, 10 Jul 2008 16:27:09 -0400
> Cc: lekktu@gmail.com, emacs-devel@gnu.org, jasonr@gnu.org,
> 	Kenichi Handa <handa@m17n.org>
> 
> > But maybe in this case we should force some safe coding like `undecided'
> > (as it seems to use now for pure ASCII Info files without null-bytes)?
> 
> No, I think it should use `us-ascii' instead.

That is almost certainly a bad idea: no need to punish users who have
non-ASCII Info files without a proper `coding' tag.  The
@documentencoding feature is relatively new in Texinfo, and it's
clumsy to use (you need to specify --enable-encoding on the command
line, or else @documentencoding is ignored), so I expect quite a few
such Info files to be out there.

> What happens if your TeXinfo file specifies a latin-1 encoding and your
> date is output in utf-8 because of your locale, then?

You get a garbled file.  In general, Texinfo's l10n features don't
work well if @documentencoding doesn't match the current locale's
encoding.

Btw, the official name of the project is Texinfo, not TeXinfo.

Re: "no-conversion" coding system

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 14 Jul 2008 21:39:35 -0400
> Cc: lekktu@gmail.com, emacs-devel@gnu.org, rms@gnu.org, handa@m17n.org
> 
> >> Its other name "binary" is a lot more unequivocal.
> > Only if you are a programmer who knows that binary files are usually
> > read with no conversions.  Otherwise, the name "binary" doesn't have
> 
> Since the `no-conversion' method depends on the underlying byte-encoding
> used in a file, it is inherently something for programmers,
> bit-twiddlers, ...

I didn't mean Lisp programmers, I meant C/C++ programmers.  Many
people program in Lisp without knowing anything about low-level OS
stuff.

Re: "no-conversion" coding system

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>     lekktu@gmail.com,
>     emacs-devel@gnu.org,
>     rms@gnu.org,
>     handa@m17n.org
> Date: Tue, 15 Jul 2008 15:34:13 +0900
> 
>  > no-conversion doesn't mean that the text will _look_ the same, it
>  > means the byte stream will be the same.
> 
> That's true as a definition, and incorrect as a matter of fact in
> multibyte buffers.  Unless Emacs enforces "no-conversion is unibyte",

It does, as a matter of fact.

> In any case, only Emacs maintainers should ever need to care about
> the representational transformations performed by coding systems

I most strongly disagree, but maybe I don't understand what you mean
by ``representational transformations''.

Re: ^M in the info files

[Stefan Monnier]

>> > But maybe in this case we should force some safe coding like `undecided'
>> > (as it seems to use now for pure ASCII Info files without null-bytes)?
>> No, I think it should use `us-ascii' instead.

> That is almost certainly a bad idea: no need to punish users who have
> non-ASCII Info files without a proper `coding' tag.

How common is that?

>> What happens if your TeXinfo file specifies a latin-1 encoding and your
>> date is output in utf-8 because of your locale, then?
> You get a garbled file.

Good, so we don't need to worry about this case since it's broken anyway.

> Btw, the official name of the project is Texinfo, not TeXinfo.

Yes, sorry, I actually know it, but old habits die hard,


        Stefan

Re: ^M in the info files

[Juanma Barranquero]

On Mon, Jul 14, 2008 at 13:44, Kenichi Handa <handa@m17n.org> wrote:

> I've just checked out the Emacs of 2008-06-03 (it doesn't
> have the above change), and still info/efag is detected as
> no-conversion.

With an up-to-date CVS Emacs, reverting the patches for revisions
1.388 and 1.389 of coding.c (and with no other change), the problem
does not happen on Windows. There are no spurious ^Ms, and info/efaq
is detected as undecided-dos.

  Juanma

Re: ^M in the info files

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: juri@jurta.org,  lekktu@gmail.com,  emacs-devel@gnu.org,  jasonr@gnu.org,  handa@m17n.org
> Date: Mon, 21 Jul 2008 00:57:02 -0400
> 
> >> > But maybe in this case we should force some safe coding like `undecided'
> >> > (as it seems to use now for pure ASCII Info files without null-bytes)?
> >> No, I think it should use `us-ascii' instead.
> 
> > That is almost certainly a bad idea: no need to punish users who have
> > non-ASCII Info files without a proper `coding' tag.
> 
> How common is that?

Quite common, especially in Latin-n world.

Re: ^M in the info files

[Stefan Monnier]

>> >> > But maybe in this case we should force some safe coding like `undecided'
>> >> > (as it seems to use now for pure ASCII Info files without null-bytes)?
>> >> No, I think it should use `us-ascii' instead.
>> 
>> > That is almost certainly a bad idea: no need to punish users who have
>> > non-ASCII Info files without a proper `coding' tag.
>> 
>> How common is that?

> Quite common, especially in Latin-n world.

If you say so.  It's defnitely not commmon in my part of the
latin-1 world.


        Stefan