set-language-environment sets default EOL type to -unix

set-language-environment sets default EOL type to -unix

[Eli Zaretskii]

While working on bug #1853, I found something that looks like a bug:
set-language-environment always produces
default-buffer-file-coding-system with -unix EOL.

This happens because set-language-environment calls
reset-language-environment, which sets default-buffer-file-coding-system
to nil.  Then set-language-environment calls
set-language-environment-coding-systems, which does this:

  (let* ((priority (get-language-info language-name 'coding-priority))
	 (default-coding (car priority))
	 (eol-type (coding-system-eol-type default-buffer-file-coding-system)))

But coding-system-eol-type interprets its argument nil as
no-conversion:

  if (NILP (coding_system))
    coding_system = Qno_conversion;

and the result is that set-language-environment-coding-systems always
attached -unix to the value of default-coding.

I think this is a bug, and I think the right fix is to change
set-language-environment-coding-systems to not pass nil to
coding-system-eol-type, but instead leave the EOL type as undecided in
this case.  Does anyone object to this fix?

Re: bug#1853: Trouble with gzipped info files on Windows

[Eli Zaretskii]

> Date: Sat, 10 Jan 2009 22:49:05 +0100
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: 
> 
> Gzipping normal CRLF info files on Windows, there's currently two problems:
> 
>   1.- For all info files:
> 
>     cd info
>     gzip efaq
>     emacs -Q --eval "(progn (set-language-environment \"UTF-8\") (info \"(efaq)\"))"
> 
>     The info pages are decoded with a -unix coding system, so lines
> contain spurious ^M characters.
> 
>     It does depend on setting the language environment (on the command
> line or .emacs). For example, with my default "Spanish" environment,
> it does not happen; but if I pass "Spanish" in the command above, the
> info pages are erroneously decoded as info-latin-1-unix.

This happened because, on DOS and Windows, file-coding-system-alist
includes the association `("" . find-buffer-file-type-coding-system)',
and find-buffer-file-type-coding-system was not ready to see an
argument whose `car' does not appear to exist because jka-compr
removed the .gz extension from its name.

I fixed find-buffer-file-type-coding-system to work correctly in this
case.

>  2.- Additionally, for info nodes that do NOT contain a Top node:
> 
>     cd info
>     emacs -Q --eval "(progn (set-language-environment \"UTF-8\") (info
> \"(ccmode)\")"
>     ;; works OK.
> 
>     gzip ccmode*
>     emacs -Q --eval "(progn (set-language-environment \"UTF-8\") (info
> \"(ccmode)\")"
>     ;;  "No such node or anchor: Top"

This is yet another separate bug: set-language-environment always sets
default-buffer-file-coding-system to *-unix.  See my other message a
few minutes ago.

Re: bug#1853: Trouble with gzipped info files on Windows

[Eli Zaretskii]

> Date: Sat, 17 Jan 2009 15:11:14 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> This happened because, on DOS and Windows, file-coding-system-alist
> includes the association `("" . find-buffer-file-type-coding-system)',
> and find-buffer-file-type-coding-system was not ready to see an
> argument whose `car' does not appear to exist because jka-compr
> removed the .gz extension from its name.

Actually, this explains why _visiting_ compressed files might produce
a buffer with ^M characters in it.  The change I made cannot help with
Info files because the buffer name is "*info*" in that case, not
FOO.gz.

After the change, visiting, e.g., emacs-1.gz produces a buffer whose
buffer-file-coding-system is -dos, and there are no ^M characters in
it.  Visiting ccmode-1.gz still produces ^M's (because ccmode-1
includes null characters in the Index node).

> >  2.- Additionally, for info nodes that do NOT contain a Top node:
> > 
> >     cd info
> >     emacs -Q --eval "(progn (set-language-environment \"UTF-8\") (info
> > \"(ccmode)\")"
> >     ;; works OK.
> > 
> >     gzip ccmode*
> >     emacs -Q --eval "(progn (set-language-environment \"UTF-8\") (info
> > \"(ccmode)\")"
> >     ;;  "No such node or anchor: Top"
> 
> This is yet another separate bug: set-language-environment always sets
> default-buffer-file-coding-system to *-unix.  See my other message a
> few minutes ago.

Specifically, I propose the change below for this latter problem.
After that change, visiting compressed files, both regular and Info
files, work correctly for me, both before or after setting a
non-default language environment.

Index: lisp/international/mule-cmds.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/international/mule-cmds.el,v
retrieving revision 1.353
diff -u -r1.353 mule-cmds.el
--- lisp/international/mule-cmds.el	9 Jan 2009 05:01:02 -0000	1.353
+++ lisp/international/mule-cmds.el	17 Jan 2009 14:01:33 -0000
@@ -1936,7 +1936,11 @@
 	 (eol-type (coding-system-eol-type default-buffer-file-coding-system)))
     (when priority
       (set-default-coding-systems
-       (if (memq eol-type '(0 1 2 unix dos mac))
+       ;; Don't use eol-type if default-buffer-file-coding-system is
+       ;; nil, because coding-system-eol-type treats nil as
+       ;; `no-conversion'.
+       (if (and default-buffer-file-coding-system
+		(memq eol-type '(0 1 2 unix dos mac)))
 	   (coding-system-change-eol-conversion default-coding eol-type)
 	 default-coding))
       (setq default-sendmail-coding-system default-coding)

Re: bug#1853: Trouble with gzipped info files on Windows

[Eli Zaretskii]

> Date: Sat, 17 Jan 2009 16:05:36 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: lekktu@gmail.com, emacs-devel@gnu.org
> 
> --- lisp/international/mule-cmds.el	9 Jan 2009 05:01:02 -0000	1.353
> +++ lisp/international/mule-cmds.el	17 Jan 2009 14:01:33 -0000
> @@ -1936,7 +1936,11 @@
>  	 (eol-type (coding-system-eol-type default-buffer-file-coding-system)))
>      (when priority
>        (set-default-coding-systems
> -       (if (memq eol-type '(0 1 2 unix dos mac))
> +       ;; Don't use eol-type if default-buffer-file-coding-system is
> +       ;; nil, because coding-system-eol-type treats nil as
> +       ;; `no-conversion'.
> +       (if (and default-buffer-file-coding-system
> +		(memq eol-type '(0 1 2 unix dos mac)))
>  	   (coding-system-change-eol-conversion default-coding eol-type)
>  	 default-coding))
>        (setq default-sendmail-coding-system default-coding)

Upon further thought, perhaps we want the default coding-systems to
have an explicit -dos EOL type on DOS and Windows, for consistency
with how we set up things on startup.  This would involve explicitly
changing eol-type of default-coding before the last line above.

WDYT?

Re: set-language-environment sets default EOL type to -unix

[Stefan Monnier]

> I think this is a bug, and I think the right fix is to change
> set-language-environment-coding-systems to not pass nil to
> coding-system-eol-type, but instead leave the EOL type as undecided in
> this case.  Does anyone object to this fix?

Your analysis sounds convincing.  If you try this change, do you indeed
get better behavior?


        Stefan

Re: set-language-environment sets default EOL type to -unix

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Kenichi Handa <handa@m17n.org>,  Juanma Barranquero <lekktu@gmail.com>,  emacs-devel@gnu.org
> Date: Sat, 17 Jan 2009 09:36:59 -0500
> 
> > I think this is a bug, and I think the right fix is to change
> > set-language-environment-coding-systems to not pass nil to
> > coding-system-eol-type, but instead leave the EOL type as undecided in
> > this case.  Does anyone object to this fix?
> 
> Your analysis sounds convincing.  If you try this change, do you indeed
> get better behavior?

Yes, the change fixes the last bit of the problem in Bug 1853.

Re: set-language-environment sets default EOL type to -unix

[Juanma Barranquero]

On Sat, Jan 17, 2009 at 16:23, Eli Zaretskii <eliz@gnu.org> wrote:

> Yes, the change fixes the last bit of the problem in Bug 1853.

Both problems, you mean?

    Juanma

Re: set-language-environment sets default EOL type to -unix

[Eli Zaretskii]

> Date: Sat, 17 Jan 2009 16:40:21 +0100
> From: Juanma Barranquero <lekktu@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org, handa@m17n.org
> 
> On Sat, Jan 17, 2009 at 16:23, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Yes, the change fixes the last bit of the problem in Bug 1853.
> 
> Both problems, you mean?

Well, there were more than just two, so ...

Re: set-language-environment sets default EOL type to -unix

[Juanma Barranquero]

On Sat, Jan 17, 2009 at 17:36, Eli Zaretskii <eliz@gnu.org> wrote:

> Well, there were more than just two, so ...

OK :-)

What I meant is that, as you said yourself, the "^M in gzipped infos"
part of the problem is not solved with this

2009-01-17  Eli Zaretskii  <eliz@gnu.org>

        * dos-w32.el (find-buffer-file-type-coding-system): If `(car
        TARGET)' does not exist, try again with its basename replaced by
        `(cdr TARGET)'.  Fixes Bug #1853.

and needs your subsequent patch or an equivalent fix.

    Juanma

Re: set-language-environment sets default EOL type to -unix

[Stefan Monnier]

>> > I think this is a bug, and I think the right fix is to change
>> > set-language-environment-coding-systems to not pass nil to
>> > coding-system-eol-type, but instead leave the EOL type as undecided in
>> > this case.  Does anyone object to this fix?
>> 
>> Your analysis sounds convincing.  If you try this change, do you indeed
>> get better behavior?

> Yes, the change fixes the last bit of the problem in Bug 1853.

Than I suggest you install it.


        Stefan

Re: bug#1853: Trouble with gzipped info files on Windows

[Stefan Monnier]

> Upon further thought, perhaps we want the default coding-systems to
> have an explicit -dos EOL type on DOS and Windows, for consistency
> with how we set up things on startup.  This would involve explicitly
> changing eol-type of default-coding before the last line above.

Ideally, it should not only do the same as done on startup, but also do
it by running the same code.


        Stefan

Re: bug#1853: Trouble with gzipped info files on Windows

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 1853@emacsbugs.donarmstrong.com,  handa@m17n.org,  lekktu@gmail.com,  emacs-devel@gnu.org
> Date: Sun, 18 Jan 2009 16:01:40 -0500
> 
> > Upon further thought, perhaps we want the default coding-systems to
> > have an explicit -dos EOL type on DOS and Windows, for consistency
> > with how we set up things on startup.  This would involve explicitly
> > changing eol-type of default-coding before the last line above.
> 
> Ideally, it should not only do the same as done on startup, but also do
> it by running the same code.

Right, I had that in mind, too.  The problem is, it's not trivial.
The way it works on startup now is a bit kludgey: dos-w32.el says

  (setq-default buffer-file-coding-system 'undecided-dos)

and then set-language-environment-coding-systems does

 (let* ((priority (get-language-info language-name 'coding-priority))
        (default-coding (car priority))
        (eol-type (coding-system-eol-type default-buffer-file-coding-system)))
  ...
    (if (memq eol-type '(0 1 2 unix dos mac))
      (coding-system-change-eol-conversion default-coding eol-type)))

the kludge being that this only works because of the specific order
files are loaded at dump time and the order of function invocation
during startup.  Are we okay with making non-trivial changes in that
at this time?

Re: bug#1853: Trouble with gzipped info files on Windows

[Stefan Monnier]

> the kludge being that this only works because of the specific order
> files are loaded at dump time and the order of function invocation
> during startup.  Are we okay with making non-trivial changes in that
> at this time?

I guess it's better to make a simpler local change, and add a big fat
comment about what should be done instead.


        Stefan

Re: bug#1853: Trouble with gzipped info files on Windows

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: 1853@emacsbugs.donarmstrong.com, handa@m17n.org, lekktu@gmail.com,
>         emacs-devel@gnu.org
> Date: Tue, 20 Jan 2009 00:10:04 -0500
> 
> > the kludge being that this only works because of the specific order
> > files are loaded at dump time and the order of function invocation
> > during startup.  Are we okay with making non-trivial changes in that
> > at this time?
> 
> I guess it's better to make a simpler local change, and add a big fat
> comment about what should be done instead.

Done, with the following change:

Index: lisp/international/mule-cmds.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/international/mule-cmds.el,v
retrieving revision 1.353
retrieving revision 1.354
diff -u -r1.353 -r1.354
--- lisp/international/mule-cmds.el	9 Jan 2009 05:01:02 -0000	1.353
+++ lisp/international/mule-cmds.el	24 Jan 2009 15:31:09 -0000	1.354
@@ -1933,7 +1933,25 @@
   "Do various coding system setups for language environment LANGUAGE-NAME."
   (let* ((priority (get-language-info language-name 'coding-priority))
 	 (default-coding (car priority))
-	 (eol-type (coding-system-eol-type default-buffer-file-coding-system)))
+	 ;; If default-buffer-file-coding-system is nil, don't use
+	 ;; coding-system-eol-type, because it treats nil as
+	 ;; `no-conversion'.  default-buffer-file-coding-system is set
+	 ;; to nil by reset-language-environment, and in that case we
+	 ;; want to have here the native EOL type for each platform.
+	 ;; FIXME: there should be a common code that runs both on
+	 ;; startup and here to set the default EOL type correctly.
+	 ;; Right now, DOS/Windows platforms set this on dos-w32.el,
+	 ;; which works only as long as the order of loading files at
+	 ;; dump time and calling functions at startup is not modified
+	 ;; significantly, i.e. as long as this function is called
+	 ;; _after_ default-buffer-file-coding-system was set by
+	 ;; dos-w32.el.
+	 (eol-type
+	  (if (null default-buffer-file-coding-system)
+	      (cond ((memq system-type '(windows-nt ms-dos)) 1)
+		    ((eq system-type 'macos) 2)
+		    (t 0))
+	    (coding-system-eol-type default-buffer-file-coding-system))))
     (when priority
       (set-default-coding-systems
        (if (memq eol-type '(0 1 2 unix dos mac))