bug#4816: change of coding system without inquiry

bug#4816: change of coding system without inquiry

[Norbert Eisinger]

When I edit a non-latin-1 character into a latin-1 file and then save it,
Emacs saves the file in UTF-8 without inquiry. It omits the dialogue that
would inform me about the offending character and would offer me to select
a coding system for saving.

Below is a description for Emacs 22.2 (Ubuntu). Other people could
reproduce the behavior in Emacs 22.1 (Ubuntu) and 23.1.

I do get the dialogue when I change the second Emacs invocation below to
~$ env LC_CTYPE=C /usr/bin/emacs -Q /tmp/l1


If suppression of the dialogue is intended and not a bug: is there
some variable with which I can specify that I want to get that
dialogue anyway?

Norbert Eisinger



=========================================================================

~$ /usr/bin/emacs -Q

,----[ C-h l ]
| <help-echo> <help-echo> ä ö ü <return> C-x <return> f i s o - l a t i n - 1 - u n i x <return> C-x C-s C-a C-k / t m p / l 1 <return> C-h l
`----

~$ file /tmp/l1
/tmp/l1: ISO-8859 text

~$ /usr/bin/emacs -Q /tmp/l1

,----[ C-h l ]
| <help-echo> M-> E u r o s i g n SPC € SPC i n s e r t e d <return> C-x C-s C-h l
`----

,----[ *Messages* ]
| ("/usr/bin/emacs" "-Q" "/tmp/l1")
| For information about GNU Emacs and the GNU system, type C-h C-a.
| Mark set
| Wrote /tmp/l1
`----

,----
| In GNU Emacs 22.2.1 (i486-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
|  of 2008-09-05 on vernadsky, modified by Ubuntu
| Windowing system distributor `The X.Org Foundation', version 11.0.10600000
| configured using `configure  '--build=i486-linux-gnu' '--host=i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs22:/etc/emacs:/usr/local/share/emacs/22.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/22.2/site-lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/22.2/leim' '--with-x=yes' '--with-x-toolkit=athena'
'--with-toolkit-scroll-bars' 'build_alias=i486-linux-gnu' 'host_alias=i486-linux-gnu' 'CFLAGS=-DDEBIAN -DSITELOAD_PURESIZE_EXTRA=5000 -g -O2' 'LDFLAGS=-g -Wl,--as-needed' 'CPPFLAGS=''
|
| Important settings:
|   value of $LC_ALL:
|   value of $LC_COLLATE: nil
|   value of $LC_CTYPE: de_DE.utf8
|   value of $LC_MESSAGES: nil
|   value of $LC_MONETARY: nil
|   value of $LC_NUMERIC: nil
|   value of $LC_TIME: de_DE.utf8
|   value of $LANG: en_US.UTF-8
|   locale-coding-system: utf-8
|   default-enable-multibyte-characters: t
|
| Major mode: Fundamental
|
| Minor modes in effect:
|   tooltip-mode: t
|   tool-bar-mode: t
|   mouse-wheel-mode: t
|   menu-bar-mode: t
|   file-name-shadow-mode: t
|   global-font-lock-mode: t
|   font-lock-mode: t
|   blink-cursor-mode: t
|   unify-8859-on-encoding-mode: t
|   utf-translate-cjk-mode: t
|   auto-compression-mode: t
|   line-number-mode: t
|   transient-mark-mode: identity
`----

~$ file /tmp/l1
/tmp/l1: UTF-8 Unicode text

=========================================================================

bug#4816: change of coding system without inquiry

[Stefan Monnier]

> When I edit a non-latin-1 character into a latin-1 file and then save it,
> Emacs saves the file in UTF-8 without inquiry. It omits the dialogue that
> would inform me about the offending character and would offer me to select
> a coding system for saving.
[...]
> If suppression of the dialogue is intended and not a bug: is there
> some variable with which I can specify that I want to get that
> dialogue anyway?

In Emacs-23, the preferred charset has been changed to utf-8, so when
opening a new file, we first try utf-8 (both because it's expected to
be the standard coding-system in GNU/Linux systems nowadays, and
because there are very few files that are encoded with something else
than utf-8 and yet they also happen to be valid utf-8 files).

So when saving a file in a latin-1 locale, both latin-1 and utf-8 can be
used "interchangeably" by Emacs.

There is currently no direct way to ask Emacs to prompt when changing
the coding system (probably mostly because changing it is common when
changing it from us-ascii (as used for empty files or other files
without non-ascii chars) to something else).

Could you explain the context in which this problem showed up?
(e.g. tell us why the coding system used matters, whether or not you've
specified the coding-system in a "-*-coding-*-" cookie and why, etc...)

BTW, in Emacs-23.2, the code has been slightly changed to always obey
the coding cookie in a file, so if you file has a "coding:latin-1"
cookie, then Emacs will not silently save it in any other coding-system.


        Stefan

bug#4816: change of coding system without inquiry

[Norbert Eisinger]

Stefan Monnier wrote:
> 
> Could you explain the context in which this problem showed up?
> (e.g. tell us why the coding system used matters, whether or not you've
> specified the coding-system in a "-*-coding-*-" cookie and why, etc...)

Cher Stefan,

we are a group in the computer science department of a German
university. The group consists of a full professor and an
assistant professor and 10 to 15 young scientists with research
duties or teaching duties or both. The young scientists typically
stay in the group for 3 or 4 years.

Our computer environment has for years been a SuSE distribution
of Linux with latin-1 locale. Recently it was replaced by a
Kubuntu distribution with utf-8 locale.

Over the years we have accumulated a large number of latin-1 files
that were created and edited by people who are no longer in the group
and who used whichever editors they liked. Some files may contain as
part of their text content information about the coding system:
"\usepackage[latin1]{inputenc}" in a LaTeX file,
"<?xml version='1.0' encoding='iso-8859-1'?>" in an XML file,
"<meta http-equiv='Content-Type' content='text/html; charset=iso-8859-1'>"
in an HTML file. Probably there are similar examples for other formats.
Depending on the format the files may also contain some form of
include mechanism that refers to a different file whose text
content contains such coding information.

The old files are still being used and edited. Sometimes people
have to make just minor modifications to an old file and do not
expect major consequences. However, if a minor modification
results in a change of the coding system, then the text content
of this file and possibly of included files may have to be adapted
in order to keep it consistent with the new actual coding system.
The colleagues are capable of repairing such inconsistencies, but
they may not notice them if the coding system changes silently.
This is why I am looking for a way to enforce a prompt whenever
the coding system is about to be changed.

In our situation there are not very few but very many files
encoded with something else than utf-8, and almost all of them
contain non-US-ASCII characters. Moreover, we never know by
which editors the files were created and subsequently edited.
Most of them do not contain "-*-coding-*-" cookies. Therefore
the new Emacs-23.2 behavior would probably not cover our case.

I don't know how frequent our situation is, but I guess that
it is not uncommon in university environments in countries
whose language needed one of the ISO-latin extensions of ASCII.


Thanks for your time and effort.

Norbert

bug#4816: change of coding system without inquiry

[Stefan Monnier]

> Our computer environment has for years been a SuSE distribution
> of Linux with latin-1 locale. Recently it was replaced by a
> Kubuntu distribution with utf-8 locale.

Fairly common situation.

> Over the years we have accumulated a large number of latin-1 files
> that were created and edited by people who are no longer in the group
> and who used whichever editors they liked. Some files may contain as
> part of their text content information about the coding system:
> "\usepackage[latin1]{inputenc}" in a LaTeX file,
> "<?xml version='1.0' encoding='iso-8859-1'?>" in an XML file,
> "<meta http-equiv='Content-Type' content='text/html; charset=iso-8859-1'>"
> in an HTML file. Probably there are similar examples for other formats.

Those xml/latex statement should be understood by Emacs as coding
cookies, so the files that contain those statements should be handled
properly with Emacs-23.2.

> Depending on the format the files may also contain some form of
> include mechanism that refers to a different file whose text
> content contains such coding information.

Yes, this is a more problematic case, because fundamentally there is
a coding-cookie, but Emacs doesn't know it because it's in another file.

> I don't know how frequent our situation is, but I guess that
> it is not uncommon in university environments in countries
> whose language needed one of the ISO-latin extensions of ASCII.

That was helpful, thank you.  I think you're right that we should at
least do a y-or-n-p prompt when changing the buffer-file-coding-system,
unless the change is to a "superset coding system" (like from us-ascii
to latin-1).

Can you try the patch below to see if it gives you the behavior you want?


        Stefan


=== modified file 'lisp/international/mule-cmds.el'
--- lisp/international/mule-cmds.el	2009-10-24 18:33:25 +0000
+++ lisp/international/mule-cmds.el	2009-10-30 15:01:02 +0000
@@ -979,6 +979,17 @@
 	(if safe
 	    (setq coding-system (car safe))))
 
+      (unless (or (local-variable-p 'buffer-file-coding-system)
+                  (eq coding-system buffer-file-coding-system)
+                  ;; We'd rather only bother the user if the coding-system
+                  ;; change would cause the file's content to change, so we'd
+                  ;; want to check whether the new coding-system is a superset
+                  ;; of the previous one.
+                  (memq buffer-file-coding-system '(nil undecided us-ascii))
+                  (y-or-n-p (format "Change encoding from %s to %s? "
+                                    buffer-file-coding-system coding-system)))
+        (setq coding-system nil))
+
       ;; If all the defaults failed, ask a user.
       (when (not coding-system)
 	(setq coding-system (select-safe-coding-system-interactively
@@ -1024,6 +1035,7 @@
 %s specified by file contents.  Really save (else edit coding cookies \
 and try again)? " coding-system auto-cs))
 	      (error "Save aborted"))))
+
       (when (and tick (/= tick (buffer-chars-modified-tick)))
 	(error "Cancelled because the buffer was modified"))
       coding-system)))

bug#4816: change of coding system without inquiry

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 30 Oct 2009 11:02:49 -0400
> Cc: 4816@emacsbugs.donarmstrong.com
> 
> +                  (eq coding-system buffer-file-coding-system)

This should probably use coding-system-equal.

bug#4816: change of coding system without inquiry

[Norbert Eisinger]

Stefan Monnier wrote:
>
> I think you're right that we should at
> least do a y-or-n-p prompt when changing the buffer-file-coding-system,
> unless the change is to a "superset coding system" (like from us-ascii
> to latin-1).

I understand the tightrope walk. When you ask users about technical stuff,
they feel bothered if it does not make any difference to them. When you
don't ask them, they feel patronized if it does make a difference.

The superset strategy is probably a reasonable compromise. I believe
that something like the following might still go wrong, but this
is a construed example and I did not test it:

File announcement.tex contains
   \include{config.tex}
   ... \LECTURER ...
   % only ASCII characters in this file

Latin-1 encoded file config.tex contains
   \usepackage[latin1]{inputenc}
   \newcommand{\LECTURER}{François}

Later someone edits announcement.tex, adds the letter ç to it and saves it,
thus changing the coding system from ascii to the superset utf-8. After that
the two occurrences of ç are differently encoded.

But, again, this is an unusual case. In normal cases we should be OK with
the superset strategy.



> There is currently no direct way to ask Emacs to prompt when changing
> the coding system

Setting select-safe-coding-system-accept-default-p to
(function (lambda (coding-system) nil))
would not have this effect in the case we are discussing, would it?



> Can you try the patch below to see if it gives you the behavior you want?

I could not apply the patch, but I am at the limit of my knowledge.
Should I have gunzipped the file first?
Below is a description of what I did.

Best regards,
Norbert


============================================================================
Local timezone: +0100

Emacs version:
GNU Emacs 22.2.1 (i486-pc-linux-gnu, X toolkit, Xaw3d scroll bars) of 2008-09-05 on vernadsky, modified by Ubuntu

$ locate mule-cmds
/usr/share/emacs/22.2/lisp/international/mule-cmds.el.gz
/usr/share/emacs/22.2/lisp/international/mule-cmds.elc
/usr/share/xemacs-21.4.21/lisp/mule/mule-cmds.elc

$ ls -l /usr/share/emacs/22.2/lisp/international/mule-cmds.*
-rw-r--r-- 1 root root  79042 2008-09-05 23:28 mule-cmds.elc
-rw-r--r-- 1 root root  30300 2008-01-10 13:15 mule-cmds.el.gz

$ cd /usr/share/emacs/22.2/lisp/international

$ /usr/bin/emacs -Q &
  copy&paste the patch into buffer *scratch*
  menu Tools -> Apply Patch -> To a File
  select buffer *scratch* and file mule-cmds.el

,----[ *scratch* ]
|
| === modified file 'lisp/international/mule-cmds.el'
| --- lisp/international/mule-cmds.el	2009-10-24 18:33:25 +0000
| +++ lisp/international/mule-cmds.el	2009-10-30 15:01:02 +0000
| @@ -979,6 +979,17 @@
|  	(if safe
|  	    (setq coding-system (car safe))))
|
| +      (unless (or (local-variable-p 'buffer-file-coding-system)
| +                  (eq coding-system buffer-file-coding-system)
| +                  ;; We'd rather only bother the user if the coding-system
| +                  ;; change would cause the file's content to change, so we'd
| +                  ;; want to check whether the new coding-system is a superset
| +                  ;; of the previous one.
| +                  (memq buffer-file-coding-system '(nil undecided us-ascii))
| +                  (y-or-n-p (format "Change encoding from %s to %s? "
| +                                    buffer-file-coding-system coding-system)))
| +        (setq coding-system nil))
| +
|        ;; If all the defaults failed, ask a user.
|        (when (not coding-system)
|  	(setq coding-system (select-safe-coding-system-interactively
| @@ -1024,6 +1035,7 @@
|  %s specified by file contents.  Really save (else edit coding cookies \
|  and try again)? " coding-system auto-cs))
|  	      (error "Save aborted"))))
| +
|        (when (and tick (/= tick (buffer-chars-modified-tick)))
|  	(error "Cancelled because the buffer was modified"))
|        coding-system)))
|
`----

,----[ *Messages* ]
| ("/usr/bin/emacs" "-Q")
| For information about GNU Emacs and the GNU system, type C-h C-a.
| Mark set
| Loading ediff...done
| Is the patch already in a buffer? (y or n)
| Making completion list...
| Loading help-mode...done
| Applying patch ... done
| ediff-patch-file-internal: Patch appears to have failed
`----

,----[ *ediff-message* ]
| Patch program has failed due to a bad patch file,
| it couldn't apply all hunks, OR
| it couldn't create the backup for the file being patched.
|
| The former could be caused by a corrupt patch file or because the "patch"
| program doesn't understand the format of the patch file in use.
|
| The second problem might be due to an incompatibility among these settings:
|     ediff-patch-program    = "patch"             ediff-patch-options    = "-f"
|     ediff-backup-extension = ".orig"             ediff-backup-specs     = "-z.orig -b"
|
| See Ediff on-line manual for more details on these variables.
| In particular, check the documentation for `ediff-backup-specs'.
|
| In any of the above cases, Ediff doesn't compare files automatically.
| However, if the patch was applied partially and the backup file was created,
| you can still examine the changes via M-x ediff-files
`----

,----[ mule-cmds.el.rej ]
| ***************
| *** 979,984 ****
|   	(if safe
|   	    (setq coding-system (car safe))))
|
|         ;; If all the defaults failed, ask a user.
|         (when (not coding-system)
|   	(setq coding-system (select-safe-coding-system-interactively
| --- 979,995 ----
|   	(if safe
|   	    (setq coding-system (car safe))))
|
| +       (unless (or (local-variable-p 'buffer-file-coding-system)
| +                   (eq coding-system buffer-file-coding-system)
| +                   ;; We'd rather only bother the user if the coding-system
| +                   ;; change would cause the file's content to change, so we'd
| +                   ;; want to check whether the new coding-system is a superset
| +                   ;; of the previous one.
| +                   (memq buffer-file-coding-system '(nil undecided us-ascii))
| +                   (y-or-n-p (format "Change encoding from %s to %s? "
| +                                     buffer-file-coding-system coding-system)))
| +         (setq coding-system nil))
| +
|         ;; If all the defaults failed, ask a user.
|         (when (not coding-system)
|   	(setq coding-system (select-safe-coding-system-interactively
| ***************
| *** 1024,1029 ****
|   %s specified by file contents.  Really save (else edit coding cookies \
|   and try again)? " coding-system auto-cs))
|   	      (error "Save aborted"))))
|         (when (and tick (/= tick (buffer-chars-modified-tick)))
|   	(error "Cancelled because the buffer was modified"))
|         coding-system)))
| --- 1035,1041 ----
|   %s specified by file contents.  Really save (else edit coding cookies \
|   and try again)? " coding-system auto-cs))
|   	      (error "Save aborted"))))
| +
|         (when (and tick (/= tick (buffer-chars-modified-tick)))
|   	(error "Cancelled because the buffer was modified"))
|         coding-system)))
|
`----

$ ls -l mule-cmds*
-rw-r--r-- 1 root root 30300 2009-10-31 15:26 mule-cmds.el
-rw-r--r-- 1 root root 79042 2008-09-05 23:28 mule-cmds.elc
-rw-r--r-- 1 root root 30300 2008-01-10 13:15 mule-cmds.el.orig
-rw-r--r-- 1 root root  1797 2009-10-31 15:26 mule-cmds.el.rej

$ diff mule-cmds.el.orig mule-cmds.el
$ ### files do not differ

$ ### lines 978 to 990 of a gunzipped safety copy made before
$ ### line number 986 seems to be the line numbered 979 in the patch
$ head -n 990 /tmp/mule-cmds-safety-copy.el | tail -n -13
        ;; Classify the defaults into safe, rejected, and unsafe.
        (dolist (elt default-coding-system)
          (if (memq (cdr elt) codings)
              (if (and (functionp accept-default-p)
                       (not (funcall accept-default-p (cdr elt))))
                  (push (car elt) rejected)
                (push (car elt) safe))
            (push (car elt) unsafe)))
        (if safe
            (setq coding-system (car safe))))

      ;; If all the defaults failed, ask a user.
      (when (not coding-system)

bug#4816: change of coding system without inquiry

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 30 Oct 2009 11:02:49 -0400
> Cc: 4816@emacsbugs.donarmstrong.com
> 
> I think you're right that we should at least do a y-or-n-p prompt
> when changing the buffer-file-coding-system, unless the change is to
> a "superset coding system" (like from us-ascii to latin-1).

FWIW, in the past users explicitly expressed annoyance by these
questions.  The request was to use the "native" encoding silently.  By
introducing back this question, we are restoring that annoyance.

bug#4816: change of coding system without inquiry

[Stefan Monnier]

>> I think you're right that we should at least do a y-or-n-p prompt
>> when changing the buffer-file-coding-system, unless the change is to
>> a "superset coding system" (like from us-ascii to latin-1).

> FWIW, in the past users explicitly expressed annoyance by these
> questions.  The request was to use the "native" encoding silently.
> By introducing back this question, we are restoring that annoyance.

IIRC, the annoyance was only when saving a formerly us-ascii file
(typical example: an empty or not-yet-existing file).
But, yes, I know this is delicate.


        Stefan

bug#4816: change of coding system without inquiry

[Lars Ingebrigtsen]

(Bug report about Latin-1 files becoming UTF-8.)

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> +      (unless (or (local-variable-p 'buffer-file-coding-system)
> +                  (eq coding-system buffer-file-coding-system)
> +                  ;; We'd rather only bother the user if the coding-system
> +                  ;; change would cause the file's content to change, so we'd
> +                  ;; want to check whether the new coding-system is a superset
> +                  ;; of the previous one.
> +                  (memq buffer-file-coding-system '(nil undecided us-ascii))
> +                  (y-or-n-p (format "Change encoding from %s to %s? "
> +                                    buffer-file-coding-system coding-system)))
> +        (setq coding-system nil))

Eli Zaretskii <eliz@gnu.org> writes:

> FWIW, in the past users explicitly expressed annoyance by these
> questions.  The request was to use the "native" encoding silently.  By
> introducing back this question, we are restoring that annoyance.

Perhaps it's time to make a decision now -- to not add a change like this?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#4816: change of coding system without inquiry

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: eisinger@informatik.uni-muenchen.de,  4816@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
> Date: Thu, 27 Jun 2019 19:49:33 +0200
> 
> > FWIW, in the past users explicitly expressed annoyance by these
> > questions.  The request was to use the "native" encoding silently.  By
> > introducing back this question, we are restoring that annoyance.
> 
> Perhaps it's time to make a decision now -- to not add a change like this?

I'm still not convinced we need this.  But if people think otherwise,
we could have an opt-in feature to ask such questions.

bug#4816: change of coding system without inquiry

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> > FWIW, in the past users explicitly expressed annoyance by these
>> > questions.  The request was to use the "native" encoding silently.  By
>> > introducing back this question, we are restoring that annoyance.
>> 
>> Perhaps it's time to make a decision now -- to not add a change like this?
>
> I'm still not convinced we need this.  But if people think otherwise,
> we could have an opt-in feature to ask such questions.

There was no response to this, so I'm guessing there wasn't much
enthusiasm for adding prompting like this, so I'm re-closing this bug
report.

And as you say:

> This is a rare use case.  It's possible that just saying
>
>   (prefer-coding-system 'latin-1)
>
> is all that's needed to get the required prompt back.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#4816: change of coding system without inquiry

[Lars Ingebrigtsen]

Norbert Eisinger <eisinger@informatik.uni-muenchen.de> writes:

> When I edit a non-latin-1 character into a latin-1 file and then save it,
> Emacs saves the file in UTF-8 without inquiry. It omits the dialogue that
> would inform me about the offending character and would offer me to select
> a coding system for saving.
>
> Below is a description for Emacs 22.2 (Ubuntu). Other people could
> reproduce the behavior in Emacs 22.1 (Ubuntu) and 23.1.

I think this has probably been fixed in the intervening years, so I'm
closing this bug report.  Please reopen if this is still an issue.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#4816: change of coding system without inquiry

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 28 Feb 2016 17:37:13 +1100
> Cc: 4816@debbugs.gnu.org
> 
> I think this has probably been fixed in the intervening years

Actually, I very much doubt that.  We still silently use the preferred
encoding if the original one doesn't fit.

This is a rare use case.  It's possible that just saying

  (prefer-coding-system 'latin-1)

is all that's needed to get the required prompt back.

I think we should also welcome a patch to provide a user option that
will always force the prompt.

bug#4816: change of coding system without inquiry

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Actually, I very much doubt that.  We still silently use the preferred
> encoding if the original one doesn't fit.
>
> This is a rare use case.  It's possible that just saying
>
>   (prefer-coding-system 'latin-1)
>
> is all that's needed to get the required prompt back.
>
> I think we should also welcome a patch to provide a user option that
> will always force the prompt.

Ok; reopening.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no