bug#64164: 29.0.92; buffer-file-coding-system changes unexpectedly after saving

bug#64164: 29.0.92; buffer-file-coding-system changes unexpectedly after saving

[miranda]

i am trying to edit remote files on another macOS system, using Tramp's 
ssh: method. i am running `emacs -Q`.

when i open any file on this remote system, the initial value of 
`buffer-file-coding-system` is what i would expect, i.e. undecided-unix 
or utf-8-unix, depending on whether the file contains non-ASCII 
characters.

however, after saving the file, `buffer-file-coding-system` suddenly 
changes to utf-8-hfs-mac. any subsequent save then changes all the line 
endings to CR, which i have not actively used since 2001 or so... :-)

i can use `set-buffer-file-coding-system` to set utf-8-unix, but the 
problem then occurs again after the next save. other remotes (Linux 
systems) do not exhibit the issue.

Emacs 28.2 works as expected.

i am using the build from https://emacsformacosx.com/builds

best,
miranda


In GNU Emacs 29.0.92 (build 1, x86_64-apple-darwin15.6.0, NS
  appkit-1404.47 Version 10.11.6 (Build 15G22010)) of 2023-06-18 built on
  builder10-11.lan
Windowing system distributor 'Apple', version 10.3.1561
System Description:  Mac OS X 10.13.6

Configured using:
  'configure --with-ns '--enable-locallisppath=/Library/Application
  Support/Emacs/${version}/site-lisp:/Library/Application
  Support/Emacs/site-lisp' --with-modules
  CFLAGS=-DNSTextAlignmentRight=NSRightTextAlignment --with-x-toolkit=no'

Configured features:
ACL GMP GNUTLS JSON LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER SQLITE3
THREADS TOOLKIT_SCROLL_BARS ZLIB

Important settings:
   value of $LANG: fi_FI.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   show-paren-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   line-number-mode: t
   indent-tabs-mode: t
   transient-mark-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/ns-win ns-win ucs-normalize mule-util term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads kqueue cocoa ns multi-tty
make-network-process emacs)

Memory information:
((conses 16 37468 8247)
  (symbols 48 5045 0)
  (strings 32 12567 2096)
  (string-bytes 1 355935)
  (vectors 16 10369)
  (vector-slots 8 155077 9845)
  (floats 8 21 23)
  (intervals 56 217 0)
  (buffers 976 10))

bug#64164: 29.0.92; buffer-file-coding-system changes unexpectedly after saving

[Eli Zaretskii]

> Date: Mon, 19 Jun 2023 12:57:39 +0300
> From: miranda@pulusound.fi
> 
> i am trying to edit remote files on another macOS system, using Tramp's 
> ssh: method. i am running `emacs -Q`.
> 
> when i open any file on this remote system, the initial value of 
> `buffer-file-coding-system` is what i would expect, i.e. undecided-unix 
> or utf-8-unix, depending on whether the file contains non-ASCII 
> characters.
> 
> however, after saving the file, `buffer-file-coding-system` suddenly 
> changes to utf-8-hfs-mac. any subsequent save then changes all the line 
> endings to CR, which i have not actively used since 2001 or so... :-)
> 
> i can use `set-buffer-file-coding-system` to set utf-8-unix, but the 
> problem then occurs again after the next save. other remotes (Linux 
> systems) do not exhibit the issue.
> 
> Emacs 28.2 works as expected.

Does this happen only with editing remote files via Tramp, or does it
also happen when you edit local files on macOS?

If it only happens with Tramp, is it possible for you to login to that
other system and edit files there locally, in case this is triggered
by something specific to that system?

> i am using the build from https://emacsformacosx.com/builds

I don't know what that is.  Are you sure this uses the official
sources from the emacs-29.0.92 tarball?

bug#64164: 29.0.92; buffer-file-coding-system changes unexpectedly after saving

[Michael Albinus]

miranda@pulusound.fi writes:

Hi Miranda,

> i am trying to edit remote files on another macOS system, using
> Tramp's ssh: method. i am running `emacs -Q`.
>
> when i open any file on this remote system, the initial value of
> `buffer-file-coding-system` is what i would expect,
> i.e. undecided-unix or utf-8-unix, depending on whether the file
> contains non-ASCII characters.
>
> however, after saving the file, `buffer-file-coding-system` suddenly
> changes to utf-8-hfs-mac. any subsequent save then changes all the
> line endings to CR, which i have not actively used since 2001 or
> so... :-)

Well, if Tramp detects a remote macOS system, it converts the coding
system to `utf-8-hfs-mac', see function
`tramp-open-connection-setup-interactive-shell'. This heuristic works
for many years w/o complaints from users.

We could suppress this. However, Tramp needs an indication when to
suppress. Do you have somthing like this?

> Emacs 28.2 works as expected.

This surprises me. I don't remember we have changed something here in
Tramp (but I might be wrong).

> best,
> miranda

Best regards, Michael.

bug#64164: 29.0.92; buffer-file-coding-system changes unexpectedly after saving

[miranda kastemaa]

[-- Attachment #1: Type: text/plain, Size: 2818 bytes --]

> 
> On 19 Jun 2023, at 19.55, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Mon, 19 Jun 2023 12:57:39 +0300
>> From: miranda@pulusound.fi
>> 
>> i am trying to edit remote files on another macOS system, using Tramp's 
>> ssh: method. i am running `emacs -Q`.
>> 
>> when i open any file on this remote system, the initial value of 
>> `buffer-file-coding-system` is what i would expect, i.e. undecided-unix 
>> or utf-8-unix, depending on whether the file contains non-ASCII 
>> characters.
>> 
>> however, after saving the file, `buffer-file-coding-system` suddenly 
>> changes to utf-8-hfs-mac. any subsequent save then changes all the line 
>> endings to CR, which i have not actively used since 2001 or so... :-)
>> 
>> i can use `set-buffer-file-coding-system` to set utf-8-unix, but the 
>> problem then occurs again after the next save. other remotes (Linux 
>> systems) do not exhibit the issue.
>> 
>> Emacs 28.2 works as expected.
> 
> Does this happen only with editing remote files via Tramp, or does it
> also happen when you edit local files on macOS?

only via Tramp.

> If it only happens with Tramp, is it possible for you to login to that
> other system and edit files there locally, in case this is triggered
> by something specific to that system?

yes, i have just now tested the same build on the system in question. with local files, the issue does not occur. but if i use Tramp/ssh: to localhost, or to a third mac, i once again get the unexpected line ending change after saving.

on this system too, 28.2 works as expected.

>> i am using the build from https://emacsformacosx.com/builds <https://emacsformacosx.com/builds>
> 
> I don't know what that is.  Are you sure this uses the official
> sources from the emacs-29.0.92 tarball?

i have not inspected the build process myself, but at https://emacsformacosx.com/about <https://emacsformacosx.com/about>, the author writes:

> The scripts I run basically just configure and build right from the GNU source—I don't add any patches or any extraneous lisp packages. I do include the old Carbon icon on the disk image because I like it better than the new Cocoa icon but it is not enabled by default.
> 
> Emacs is built on various versions of Mac OS X: 10.10 and 10.14 as of 2020-05-12 (64 bit only). All the binaries are combined into a single executable and a small Rust launcher chooses which binary to run based on the machine's OS and architecture.
> 
> Why not just use a fat binary? Because fat binaries can only hold 1 of each architecture and Emacs has multiple x86_64 architectures binaries.
> 
> Why are there multiple x86_64 binaries? Even though recent versions of Emacs contain runtime feature detection, there is an issue with some library dependencies.

best,
miranda

[-- Attachment #2: Type: text/html, Size: 10620 bytes --]

bug#64164: 29.0.92; buffer-file-coding-system changes unexpectedly after saving

[miranda kastemaa]

19 Jun 2023 20.36.06 Michael Albinus <michael.albinus@gmx.de>:

> miranda@pulusound.fi writes:
>
> Hi Miranda,
>
>> i am trying to edit remote files on another macOS system, using
>> Tramp's ssh: method. i am running `emacs -Q`.
>>
>> when i open any file on this remote system, the initial value of
>> `buffer-file-coding-system` is what i would expect,
>> i.e. undecided-unix or utf-8-unix, depending on whether the file
>> contains non-ASCII characters.
>>
>> however, after saving the file, `buffer-file-coding-system` suddenly
>> changes to utf-8-hfs-mac. any subsequent save then changes all the
>> line endings to CR, which i have not actively used since 2001 or
>> so... :-)
>
> Well, if Tramp detects a remote macOS system, it converts the coding
> system to `utf-8-hfs-mac', see function
> `tramp-open-connection-setup-interactive-shell'. This heuristic works
> for many years w/o complaints from users.

thank you for pointing me towards this function. it is late here but this will help me do some more digging tomorrow.

> We could suppress this. However, Tramp needs an indication when to
> suppress. Do you have somthing like this?

without knowing anything about how the current logic came to be, it seems to me that -mac should never be the default on Darwin-based macOS, as the CR file endings fell out of use with the discontinuation of the "classic" Mac OS that predated Darwin.

it is perhaps worth reiterating that the files i have been editing already contain LF file endings exclusively (and thus far Emacs has always created these by default, both locally and over Tramp), so the sudden unprompted change to CR is quite disruptive. even if CR were a reasonable default, i believe Emacs is meant to respect the existing coding system of files, so this definitely seems like a bug.

>> Emacs 28.2 works as expected.
>
> This surprises me. I don't remember we have changed something here in
> Tramp (but I might be wrong).
>
>> best,
>> miranda
> > Best regards, Michael.

best,
miranda

bug#64164: 29.0.92; buffer-file-coding-system changes unexpectedly after saving

[miranda]

On 2023-06-20 01:29, miranda kastemaa wrote:
> 19 Jun 2023 20.36.06 Michael Albinus <michael.albinus@gmx.de>:
>> Well, if Tramp detects a remote macOS system, it converts the coding
>> system to `utf-8-hfs-mac', see function
>> `tramp-open-connection-setup-interactive-shell'. This heuristic works
>> for many years w/o complaints from users.
> 
> thank you for pointing me towards this function. it is late here but
> this will help me do some more digging tomorrow.

after looking at `tramp-open-connection-setup-interactive-shell` a 
little, i wonder whether the coding system here is the relevant one. if 
i understand correctly, this seems to be setting the coding system for 
the connection process, rather than the individual file buffers. on both 
29.0.92 and 28.2, the Tramp debug output here is:

> tramp-open-connection-setup-interactive-shell (5) # Setting coding 
> system to ‘utf-8-hfs’ and ‘utf-8-hfs-mac’

neither of which is a -unix coding system as (initially) reported by 
`buffer-file-coding-system`.

opening files produces more debug output, but i failed to find any more 
salient info in it (i tried searching for "coding", "utf", "-unix", 
"-mac" etc).

let me know if you have any further suggestions for debugging.

best,
miranda

bug#64164: 29.0.92; buffer-file-coding-system changes unexpectedly after saving

[Michael Albinus]

miranda@pulusound.fi writes:

Hi Miranda,

> let me know if you have any further suggestions for debugging.

I was quiet for a while because I couldn't give you further
advice. Since the problem happened on macOS, which I don't use, I
couldn't debug.

Fortunately, there's another bug report bug#65022, which looks very
similar to your problem. Here I could debug. Could you please check,
whether the patch I have proposed in
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=65022#8> helps you?

> best,
> miranda

Best regards, Michael.

bug#64164: 29.0.92; buffer-file-coding-system changes unexpectedly after saving

[Michael Albinus]

Version: 29.2

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Miranda,

> Fortunately, there's another bug report bug#65022, which looks very
> similar to your problem. Here I could debug. Could you please check,
> whether the patch I have proposed in
> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=65022#8> helps you?

No further comment, so I assume it is working. Closing the bug.

Best regards, Michael.