problem with system_eol_type

problem with system_eol_type

[Kenichi Handa]

I've got several complaints about the change I made a few
months ago regarding the handling of the default eol-type.

Previously, when a coding system without explicit eol-type
(e.g. iso-latin-1) was specified for encoding, Unix-like
eol-type is selected on any platform.

The change I made was to use an eol-type set to
system_eol_type (CRLF on Windows, LF otherwise) in such a
case.  To me, that change was just a bug fix.

But, the bug-reports say that there are many codes that
assumes the previous behaviour, and some of them now don't
work well on Windows.

I think that it is cleaner to ask a program to specify the
eol-type explicitly if it requires a specific eol-type on
encoding.  But, as long as the behavior is clearly
documented, it seems that the previous behaviour is also not
that bad.

What do people think?

---
Kenichi Handa
handa@m17n.org

PS.  I included mule-ja maining list in CC:.  Mule-ja
subscribers please write in English if you want to reply to
this mail.

Re: problem with system_eol_type

[Reiner Steib]

On Mon, Jul 31 2006, Kenichi Handa wrote:

> I've got several complaints about the change I made a few
> months ago regarding the handling of the default eol-type.
>
> Previously, when a coding system without explicit eol-type
> (e.g. iso-latin-1) was specified for encoding, Unix-like
> eol-type is selected on any platform.
>
> The change I made was to use an eol-type set to
> system_eol_type (CRLF on Windows, LF otherwise) in such a
> case.  To me, that change was just a bug fix.
>
> But, the bug-reports say that there are many codes that
> assumes the previous behaviour, and some of them now don't
> work well on Windows.

FWIW, two examples in Gnus where that doubled newlines appeared in
mail messages and NOV (overview) files weren't read/written correctly
anymore, see this discussion on the Gnus list:
<http://thread.gmane.org/gmane.emacs.gnus.general/63496/focus=63502>

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: problem with system_eol_type

[Stefan Monnier]

>>>>> "Reiner" == Reiner Steib <reinersteib+gmane@imap.cc> writes:

> On Mon, Jul 31 2006, Kenichi Handa wrote:
>> I've got several complaints about the change I made a few
>> months ago regarding the handling of the default eol-type.
>> 
>> Previously, when a coding system without explicit eol-type
>> (e.g. iso-latin-1) was specified for encoding, Unix-like
>> eol-type is selected on any platform.
>> 
>> The change I made was to use an eol-type set to
>> system_eol_type (CRLF on Windows, LF otherwise) in such a
>> case.  To me, that change was just a bug fix.
>> 
>> But, the bug-reports say that there are many codes that
>> assumes the previous behaviour, and some of them now don't
>> work well on Windows.

I'd argue that those pieces of code had bugs: if your code depends on a Unix
style EOL, it should say so explicitly.

> FWIW, two examples in Gnus where that doubled newlines appeared in
> mail messages and NOV (overview) files weren't read/written correctly
> anymore, see this discussion on the Gnus list:
> <http://thread.gmane.org/gmane.emacs.gnus.general/63496/focus=63502>

Is there any good reason why Gnus uses coding systems of the form `foo'
rather than `foo-unix'?

This said, there's clearly a problem of backward compatibility since the
"buggy" code worked in Emacs-21.


        Stefan

Re: problem with system_eol_type

[Jason Rumney]

Stefan Monnier wrote:
> Is there any good reason why Gnus uses coding systems of the form `foo'
> rather than `foo-unix'?
>
> This said, there's clearly a problem of backward compatibility since the
> "buggy" code worked in Emacs-21.
>   

Gnus is released with Emacs, so this needn't be a concern if it is fixed 
in the Gnus source before release.

The question is, do any other packages rely on the default eol-type for 
encoding being Unix? For most uses, it shouldn't matter, it is only when 
encoding to a string rather than a file that there is a problem if I 
understand the Gnus case correctly.

Re: problem with system_eol_type

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Mon, 31 Jul 2006 15:04:59 +0900
> Cc: mule-ja@m17n.org
> 
> But, the bug-reports say that there are many codes that
> assumes the previous behaviour, and some of them now don't
> work well on Windows.

What other packages, besides Gnus, were broken by this change?

> I think that it is cleaner to ask a program to specify the
> eol-type explicitly if it requires a specific eol-type on
> encoding.

I agree.

Re: problem with system_eol_type

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 31 Jul 2006 12:30:26 -0400
> Cc: mule-ja@m17n.org, emacs-devel@gnu.org
> 
> I'd argue that those pieces of code had bugs: if your code depends on a Unix
> style EOL, it should say so explicitly.

100% agreement.

Re: problem with system_eol_type

[Eli Zaretskii]

> Date: Mon, 31 Jul 2006 18:00:52 +0100
> From: Jason Rumney <jasonr@gnu.org>
> Cc: emacs-devel@gnu.org, mule-ja@m17n.org, Kenichi Handa <handa@m17n.org>
> 
> Stefan Monnier wrote:
> > Is there any good reason why Gnus uses coding systems of the form `foo'
> > rather than `foo-unix'?
> >
> > This said, there's clearly a problem of backward compatibility since the
> > "buggy" code worked in Emacs-21.
> >   
> 
> Gnus is released with Emacs, so this needn't be a concern if it is fixed 
> in the Gnus source before release.

Agreed.

Re: problem with system_eol_type

[Stefan Monnier]

> Is there any good reason why Gnus uses coding systems of the form `foo'
> rather than `foo-unix'?

More to the point: is there any reason why any package should ever use
`raw-text' (rather than, say, `binary' aka `raw-text-unix')?


        Stefan

Re: problem with system_eol_type

[Richard Stallman]

    Previously, when a coding system without explicit eol-type
    (e.g. iso-latin-1) was specified for encoding, Unix-like
    eol-type is selected on any platform.

    The change I made was to use an eol-type set to
    system_eol_type (CRLF on Windows, LF otherwise) in such a
    case.  To me, that change was just a bug fix.

This is clearly The Right Thing, at the user level.

    But, the bug-reports say that there are many codes that
    assumes the previous behaviour, and some of them now don't
    work well on Windows.

Thus, what's good at the UI level is not good at the API level.
Programs would usually like to get uniform results.

I see two possible approaches to this:

1. Fix those programs to specify `-unix'.  That way, they should get
LF eol behavior in all Emacs versions on all systems.

2. Take out your old change, and implement something at the UI level,
perhaps in a function that reads a coding system name.

#1 is clearly easier.  Is there any strong argument against it?


Is there a way 

Re: problem with system_eol_type

[Katsumi Yamaoka]

;; Mule-ja people, you can read all the followups at:
;; http://news.gmane.org/group/gmane.emacs.devel/thread=57837/force_load=t

>>>>> In <E1G7g47-00074N-5S@fencepost.gnu.org> Richard Stallman wrote:

[...]

> I see two possible approaches to this:

> 1. Fix those programs to specify `-unix'.  That way, they should get
> LF eol behavior in all Emacs versions on all systems.

> 2. Take out your old change, and implement something at the UI level,
> perhaps in a function that reads a coding system name.

> #1 is clearly easier.  Is there any strong argument against it?

I think it is reasonable to use the eol which is system's
default when saving text files.  Since Gnus uses `raw-text' by
default to save files in many cases, mail files, for example,
are now saved with CRLF in DOS machines, otherwise with LF.  It
will be useful when other programs use those files.  So, I agree
to the recent behavior of Emacs but am negative to change Gnus'
default.

By the way, why is not CR used in Mac when specifying the coding
system `foo', not `foo-eol'?

Contrary to this argument concerning saving of files, I hope the
forms

(encode-coding-string "bar\n" 'foo)

and

(with-temp-buffer
  (insert "bar\n")
  (encode-coding-region (point-min) (point-max) 'foo)
  (buffer-string))

always return "bar\n" regardless of the system-type in the
future as well.

I don't have an idea how we should do when communicating with
processes.  Sorry.

Re: problem with system_eol_type

[Stefan Monnier]

> By the way, why is not CR used in Mac when specifying the coding
> system `foo', not `foo-eol'?

Only Mac OS 9 used CR-eol.  Mac OS X uses LF, like any other Unix variant.

> Contrary to this argument concerning saving of files, I hope the
> forms

> (encode-coding-string "bar\n" 'foo)

> and

> (with-temp-buffer
>   (insert "bar\n")
>   (encode-coding-region (point-min) (point-max) 'foo)
>   (buffer-string))

> always return "bar\n" regardless of the system-type in the
> future as well.

Why this discrepancy?


        Stefan

Re: problem with system_eol_type

[Katsumi Yamaoka]

>>>>> In <jwvfyghi1od.fsf-monnier+emacs@gnu.org> Stefan Monnier wrote:

>> By the way, why is not CR used in Mac when specifying the coding
>> system `foo', not `foo-eol'?

> Only Mac OS 9 used CR-eol.  Mac OS X uses LF, like any other Unix variant.

Thanks.  It backs up the validity of the recent change in the
nnheader-insert-head function.

>> Contrary to this argument concerning saving of files, I hope the
>> forms

>> (encode-coding-string "bar\n" 'foo)

>> and

>> (with-temp-buffer
>>   (insert "bar\n")
>>   (encode-coding-region (point-min) (point-max) 'foo)
>>   (buffer-string))

>> always return "bar\n" regardless of the system-type in the
>> future as well.

> Why this discrepancy?

Because (at least) Gnus will need to be fixed here and there if
Emacs is changed so as to return "bar\r\n" or "bar\r" for those
forms.  For instance, if there are Latin-1 characters in a
message body, "bar\n" will be encoded by quoted-printable into
"bar=0D\n" or "bar=0D", which is not a good result.  I think it
is reasonable that LF is the default eol for strings and
buffer's contents since they are normally used internally in
Emacs.

Re: problem with system_eol_type

[Katsumi Yamaoka]

;; To see the whole thread, please visit:
;; http://news.gmane.org/group/gmane.emacs.devel/thread=57837/force_load=t

>>>>> In <b4mbqr5xled.fsf@jpl.org> Katsumi Yamaoka wrote:

> I think it is reasonable to use the eol which is system's
> default when saving text files.  Since Gnus uses `raw-text' by
> default to save files in many cases, mail files, for example,
> are now saved with CRLF in DOS machines, otherwise with LF.  It
> will be useful when other programs use those files.  So, I agree
> to the recent behavior of Emacs but am negative to change Gnus'
> default.

[...]

> I don't have an idea how we should do when communicating with
> processes.  Sorry.

In relation to this, ARISAWA Akihiro reported that the compface.el
module bundled with Gnus doesn't work on Windows.  It is because
the line-break code is converted into CRLF by the recent Emacs 22
when communicating with processes, and at least the external
"icontopbm" program does not work with it.

I've fixed it in the Gnus trunk and the v5-10 branch so that
`coding-system-for-write' is bound to `raw-text-unix'.

Regards,

Re: problem with system_eol_type

[Stefan Monnier]

> I've fixed it in the Gnus trunk and the v5-10 branch so that
> `coding-system-for-write' is bound to `raw-text-unix'.

Again, may I ask why you chose raw-text-unix rather than the more obviously
correct `binary'?


        Stefan

Re: problem with system_eol_type

[Katsumi Yamaoka]

>>>>> Stefan Monnier wrote:

>> I've fixed it in the Gnus trunk and the v5-10 branch so that
>> `coding-system-for-write' is bound to `raw-text-unix'.

> Again, may I ask why you chose raw-text-unix rather than the more obviously
> correct `binary'?

There was no other purpose to have used raw-text-unix.  I've
changed it to binary now.  Thanks.

Re: problem with system_eol_type

[Richard Stallman]

    In relation to this, ARISAWA Akihiro reported that the compface.el
    module bundled with Gnus doesn't work on Windows.  It is because
    the line-break code is converted into CRLF by the recent Emacs 22
    when communicating with processes, and at least the external
    "icontopbm" program does not work with it.

    I've fixed it in the Gnus trunk and the v5-10 branch so that
    `coding-system-for-write' is bound to `raw-text-unix'.

Whatever the right fix is, can you fix it in the Emacs repository too,
please?

Re: problem with system_eol_type

[Reiner Steib]

On Thu, Aug 10 2006, Richard Stallman wrote:

>     I've fixed it in the Gnus trunk and the v5-10 branch so that
>     `coding-system-for-write' is bound to `raw-text-unix'.
>
> Whatever the right fix is, can you fix it in the Emacs repository too,
> please?

Changes from the stable Gnus branch (v5-10) are (semi-) automatically
propagated to the Emacs repository by Miles Bader (using his arch
repositories).  Usually the delay is only a few days.  (Similar for
the other direction.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/