on ESHELL, utf-8 and fossil command-line commit message

on ESHELL, utf-8 and fossil command-line commit message

[Wayne Harris via Users list for the GNU Emacs text editor]

As far as I can tell in my GNU EMACS, all by buffers relevant to context
here are set to UTF-8.  I see the letter U in the first letter of the
modeline --- the GNU EMACS status bar.  (I'm on Windows, by the way.)

When I add an UTF-8 message through the command line in ESHELL, it
doesn't display properly when I say ``fossil timeline''.  Check it:

--8<---------------cut here---------------start------------->8---
%fs add encoding.txt 
ADDED  encoding.txt

%fs commit -m 'Naïve commit.'
Pull from https://mer@somewhere.edu/test
Round-trips: 1   Artifacts sent: 0  received: 0
Pull done, wire bytes sent: 442  received: 1959  ip: 5.161.138.46
New_Version: f4c20ecefc9d04d5fd2548eb4d3008d9ffb759a4deebedd595b222f81eef6b1f
Sync with https://mer@somewhere.edu/test
Round-trips: 1   Artifacts sent: 2  received: 0
Sync done, wire bytes sent: 2342  received: 309  ip: 5.161.138.46

%fs timeline
=== 2022-10-01 ===
14:03:28 [f4c20ecefc] *CURRENT* Naïve commit. (user: mer tags: trunk)
[...]
--8<---------------cut here---------------end--------------->8---

However, if instead of the command-line, I use a regular GNU EMACS
buffer, it works just fine.

--8<---------------cut here---------------start------------->8---
%echo kkk >> encoding.txt 

%fs commit
Pull from https://mer@somewhere.edu/test
Round-trips: 1   Artifacts sent: 0  received: 0
Pull done, wire bytes sent: 437  received: 2118  ip: 5.161.138.46
emacsclientw ./ci-comment-A2803F45F10B.txt
Waiting for Emacs...
Pull from https://mer@somewhere.edu/test
Round-trips: 1   Artifacts sent: 0  received: 0
Pull done, wire bytes sent: 441  received: 2118  ip: 5.161.138.46
New_Version: 09ea1b5d5b8d776d61a74bb412cd58bd8b6f82323c2f539a1eb0d915f7026f20
Sync with https://mer@somewhere.edu/test
Round-trips: 1   Artifacts sent: 2  received: 0
Sync done, wire bytes sent: 2496  received: 309  ip: 5.161.138.46

%fs timeline
=== 2022-10-01 ===
14:09:39 [09ea1b5d5b] *CURRENT* Naiveté. (user: mer tags: trunk)
--8<---------------cut here---------------end--------------->8---

Who is mangling my command-line arguments and why?  Thank you!

Re: on ESHELL, utf-8 and fossil command-line commit message

[Eli Zaretskii]

> Date: Sat, 01 Oct 2022 11:19:43 -0300
> From:  Wayne Harris via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>
> 
> As far as I can tell in my GNU EMACS, all by buffers relevant to context
> here are set to UTF-8.  I see the letter U in the first letter of the
> modeline --- the GNU EMACS status bar.  (I'm on Windows, by the way.)

Bad idea.  See below.

> When I add an UTF-8 message through the command line in ESHELL, it
> doesn't display properly when I say ``fossil timeline''.  Check it:
> 
> --8<---------------cut here---------------start------------->8---
> %fs add encoding.txt 
> ADDED  encoding.txt
> 
> %fs commit -m 'Naïve commit.'
> Pull from https://mer@somewhere.edu/test
> Round-trips: 1   Artifacts sent: 0  received: 0
> Pull done, wire bytes sent: 442  received: 1959  ip: 5.161.138.46
> New_Version: f4c20ecefc9d04d5fd2548eb4d3008d9ffb759a4deebedd595b222f81eef6b1f
> Sync with https://mer@somewhere.edu/test
> Round-trips: 1   Artifacts sent: 2  received: 0
> Sync done, wire bytes sent: 2342  received: 309  ip: 5.161.138.46
> 
> %fs timeline
> === 2022-10-01 ===
> 14:03:28 [f4c20ecefc] *CURRENT* Naïve commit. (user: mer tags: trunk)
> [...]
> --8<---------------cut here---------------end--------------->8---
> 
> However, if instead of the command-line, I use a regular GNU EMACS
> buffer, it works just fine.
> 
> --8<---------------cut here---------------start------------->8---
> %echo kkk >> encoding.txt 
> 
> %fs commit
> Pull from https://mer@somewhere.edu/test
> Round-trips: 1   Artifacts sent: 0  received: 0
> Pull done, wire bytes sent: 437  received: 2118  ip: 5.161.138.46
> emacsclientw ./ci-comment-A2803F45F10B.txt
> Waiting for Emacs...
> Pull from https://mer@somewhere.edu/test
> Round-trips: 1   Artifacts sent: 0  received: 0
> Pull done, wire bytes sent: 441  received: 2118  ip: 5.161.138.46
> New_Version: 09ea1b5d5b8d776d61a74bb412cd58bd8b6f82323c2f539a1eb0d915f7026f20
> Sync with https://mer@somewhere.edu/test
> Round-trips: 1   Artifacts sent: 2  received: 0
> Sync done, wire bytes sent: 2496  received: 309  ip: 5.161.138.46
> 
> %fs timeline
> === 2022-10-01 ===
> 14:09:39 [09ea1b5d5b] *CURRENT* Naiveté. (user: mer tags: trunk)
> --8<---------------cut here---------------end--------------->8---
> 
> Who is mangling my command-line arguments and why?  Thank you!

You cannot use UTF-8 for encoding command-line arguments of
subprocesses on MS-Windows.  Use your system's codepage instead.  This
means make sure the Eshell buffer's value of buffer-file-coding-system
is your system codepage, not UTF-8.

Re: on ESHELL, utf-8 and fossil command-line commit message

[Wayne Harris via Users list for the GNU Emacs text editor]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 01 Oct 2022 11:19:43 -0300
>> From:  Wayne Harris via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>
>> 
>> As far as I can tell in my GNU EMACS, all by buffers relevant to context
>> here are set to UTF-8.  I see the letter U in the first letter of the
>> modeline --- the GNU EMACS status bar.  (I'm on Windows, by the way.)
>
> Bad idea.  See below.
>
>> When I add an UTF-8 message through the command line in ESHELL, it
>> doesn't display properly when I say ``fossil timeline''.  Check it:
>> 
>> --8<---------------cut here---------------start------------->8---
>> %fs add encoding.txt 
>> ADDED  encoding.txt
>> 
>> %fs commit -m 'Naïve commit.'
>> Pull from https://mer@somewhere.edu/test
>> Round-trips: 1   Artifacts sent: 0  received: 0
>> Pull done, wire bytes sent: 442  received: 1959  ip: 5.161.138.46
>> New_Version: f4c20ecefc9d04d5fd2548eb4d3008d9ffb759a4deebedd595b222f81eef6b1f
>> Sync with https://mer@somewhere.edu/test
>> Round-trips: 1   Artifacts sent: 2  received: 0
>> Sync done, wire bytes sent: 2342  received: 309  ip: 5.161.138.46
>> 
>> %fs timeline
>> === 2022-10-01 ===
>> 14:03:28 [f4c20ecefc] *CURRENT* Naïve commit. (user: mer tags: trunk)
>> [...]
>> --8<---------------cut here---------------end--------------->8---
>> 
>> However, if instead of the command-line, I use a regular GNU EMACS
>> buffer, it works just fine.
>> 
>> --8<---------------cut here---------------start------------->8---
>> %echo kkk >> encoding.txt 
>> 
>> %fs commit
>> Pull from https://mer@somewhere.edu/test
>> Round-trips: 1   Artifacts sent: 0  received: 0
>> Pull done, wire bytes sent: 437  received: 2118  ip: 5.161.138.46
>> emacsclientw ./ci-comment-A2803F45F10B.txt
>> Waiting for Emacs...
>> Pull from https://mer@somewhere.edu/test
>> Round-trips: 1   Artifacts sent: 0  received: 0
>> Pull done, wire bytes sent: 441  received: 2118  ip: 5.161.138.46
>> New_Version: 09ea1b5d5b8d776d61a74bb412cd58bd8b6f82323c2f539a1eb0d915f7026f20
>> Sync with https://mer@somewhere.edu/test
>> Round-trips: 1   Artifacts sent: 2  received: 0
>> Sync done, wire bytes sent: 2496  received: 309  ip: 5.161.138.46
>> 
>> %fs timeline
>> === 2022-10-01 ===
>> 14:09:39 [09ea1b5d5b] *CURRENT* Naiveté. (user: mer tags: trunk)
>> --8<---------------cut here---------------end--------------->8---
>> 
>> Who is mangling my command-line arguments and why?  Thank you!
>
> You cannot use UTF-8 for encoding command-line arguments of
> subprocesses on MS-Windows.  Use your system's codepage instead.  This
> means make sure the Eshell buffer's value of buffer-file-coding-system
> is your system codepage, not UTF-8.

I'm glad I added that parenthesis about Windows.  (Thank you for the info!)

However, I still did not manage to get this right.  Could it be that I
don't know what my codepage is?  When I run ``chcp'', I get 850.

--8<---------------cut here---------------start------------->8---
C:\Users\mer>chcp
Active code page: 850
--8<---------------cut here---------------end--------------->8---

Then in ESHELL I type:

--8<---------------cut here---------------start------------->8---
%(setq buffer-file-coding-system 'cp850-dos)
cp850-dos

%echo naive >> encoding.txt 

%fs commit -m 'Naïve.'
[...]
Sync done, wire bytes sent: 2882  received: 309  ip: 5.161.138.46

%fs timeline
=== 2022-10-01 ===
18:26:22 [c7835d354b] *CURRENT* Naïve. (user: mer tags: trunk)
[...]
New_Version: c7835d354b[...]
%
--8<---------------cut here---------------end--------------->8---

Re: on ESHELL, utf-8 and fossil command-line commit message

[Eli Zaretskii]

> Date: Sat, 01 Oct 2022 15:29:58 -0300
> From:  Wayne Harris via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>
> 
> > You cannot use UTF-8 for encoding command-line arguments of
> > subprocesses on MS-Windows.  Use your system's codepage instead.  This
> > means make sure the Eshell buffer's value of buffer-file-coding-system
> > is your system codepage, not UTF-8.
> 
> I'm glad I added that parenthesis about Windows.  (Thank you for the info!)
> 
> However, I still did not manage to get this right.  Could it be that I
> don't know what my codepage is?  When I run ``chcp'', I get 850.
> 
> --8<---------------cut here---------------start------------->8---
> C:\Users\mer>chcp
> Active code page: 850
> --8<---------------cut here---------------end--------------->8---

Unlikely.  What does this say in Emacs:

  M-: w32-ansi-code-page RET

> Then in ESHELL I type:
> 
> --8<---------------cut here---------------start------------->8---
> %(setq buffer-file-coding-system 'cp850-dos)
> cp850-dos
> 
> %echo naive >> encoding.txt 
> 
> %fs commit -m 'Naïve.'
> [...]
> Sync done, wire bytes sent: 2882  received: 309  ip: 5.161.138.46
> 
> %fs timeline
> === 2022-10-01 ===
> 18:26:22 [c7835d354b] *CURRENT* Naïve. (user: mer tags: trunk)
> [...]
> New_Version: c7835d354b[...]
> %
> --8<---------------cut here---------------end--------------->8---

Maybe you have other customizations that cause this.  What does the
below say:

  M-: default-process-coding-system RET

Re: on ESHELL, utf-8 and fossil command-line commit message

[Wayne Harris via Users list for the GNU Emacs text editor]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 01 Oct 2022 15:29:58 -0300
>> From:  Wayne Harris via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>
>> 
>> > You cannot use UTF-8 for encoding command-line arguments of
>> > subprocesses on MS-Windows.  Use your system's codepage instead.  This
>> > means make sure the Eshell buffer's value of buffer-file-coding-system
>> > is your system codepage, not UTF-8.
>> 
>> I'm glad I added that parenthesis about Windows.  (Thank you for the info!)
>> 
>> However, I still did not manage to get this right.  Could it be that I
>> don't know what my codepage is?  When I run ``chcp'', I get 850.
>> 
>> --8<---------------cut here---------------start------------->8---
>> C:\Users\mer>chcp
>> Active code page: 850
>> --8<---------------cut here---------------end--------------->8---
>
> Unlikely.  What does this say in Emacs:
>
>   M-: w32-ansi-code-page RET

It says 1252:

%(print w32-ansi-code-page)
1252
%

My new attempt:

--8<---------------cut here---------------start------------->8---
%(setq buffer-file-coding-system 'cp1252)
cp1252

%fs commit -m 'Naiveté.'

%fs timeline -n 1
=== 2022-10-01 ===
19:02:24 [0940d7ce5e] *CURRENT* Naiveté. (user: mer tags: trunk)
--- entry limit (1) reached ---
%
--8<---------------cut here---------------end--------------->8---

When I setq buffer-[...], I saw the modeline changing the encoding to a
'*'.  Hovering the mouse, I saw 

  Buffer coding system (multi-byte): cp1252, [...]

Mouse-3-clicking the '*', I see:

--8<---------------cut here---------------start------------->8---
* -- cp1252 (alias of windows-1252)

windows-1252 (Western European) encoding (MIME: WINDOWS-1252)
Type: charset (charset)
EOL type: Automatic selection from:
	[windows-1252-unix windows-1252-dos windows-1252-mac]
This coding system encodes the following charsets:
  windows-1252
--8<---------------cut here---------------end--------------->8---

>> Then in ESHELL I type:
>> 
>> --8<---------------cut here---------------start------------->8---
>> %(setq buffer-file-coding-system 'cp850-dos)
>> cp850-dos
>> 
>> %echo naive >> encoding.txt 
>> 
>> %fs commit -m 'Naïve.'
>> [...]
>> Sync done, wire bytes sent: 2882  received: 309  ip: 5.161.138.46
>> 
>> %fs timeline
>> === 2022-10-01 ===
>> 18:26:22 [c7835d354b] *CURRENT* Naïve. (user: mer tags: trunk)
>> [...]
>> New_Version: c7835d354b[...]
>> %
>> --8<---------------cut here---------------end--------------->8---
>
> Maybe you have other customizations that cause this.  What does the
> below say:
>
>   M-: default-process-coding-system RET

It says 

%(print default-process-coding-system)
(utf-8-dos . utf-8-unix)
%

I tried to change it to just cp1252.

--8<---------------cut here---------------start------------->8---
%(setq default-process-coding-system (cons 'cp1252 'cp1252))

%echo hello >> encoding.txt 

%fs commit -m 'Naiveté.'
[...]
New_Version: d992644b4bd515df3b9b99cd978e9081ea775098398f5e83cc359967dff470c3
[...]
Sync done, wire bytes sent: 3147  received: 309  ip: 5.161.138.46

%fs timeline -n 1
=== 2022-10-01 ===
19:04:34 [d992644b4b] *CURRENT* Naiveté. (user: mer tags: trunk)
--- entry limit (1) reached ---
%
--8<---------------cut here---------------end--------------->8---

What do you think?

Re: on ESHELL, utf-8 and fossil command-line commit message

[Eli Zaretskii]

> Date: Sat, 01 Oct 2022 16:09:41 -0300
> From:  Wayne Harris via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>
> 
> > Maybe you have other customizations that cause this.  What does the
> > below say:
> >
> >   M-: default-process-coding-system RET
> 
> It says 
> 
> %(print default-process-coding-system)
> (utf-8-dos . utf-8-unix)
> %
> 
> I tried to change it to just cp1252.
> 
> --8<---------------cut here---------------start------------->8---
> %(setq default-process-coding-system (cons 'cp1252 'cp1252))

Too late, I think.

So I think the problem is that your customizations set up UTF-8
everywhere, and that causes the problems.  My suggestion is to start
from "emacs -Q", and if these commands work there, review your
customizations until you find those which get in the way.

If even "emacs -Q" doesn't work, I suggest to submit a bug report with
all the details.

In general, in core we don't pass arbitrary text via command-line
arguments on MS-Windows; instead, we write text to a temporary file
and ask the program to read text from there -- for this very reason.
Most VCS commands accept -f or -F switch telling them to read the log
message from a file; I suggest to use that instead of fighting the
UTF-8 uphill battle.

> %fs timeline -n 1
> === 2022-10-01 ===
> 19:04:34 [d992644b4b] *CURRENT* Naiveté. (user: mer tags: trunk)
> --- entry limit (1) reached ---

Could it be that 'fs' expects UTF-8?  Is that a native Windows port or
a Cygwin port?  The latter will probably expect UTF-8 on the command
line, and you cannot give it that.

Re: on ESHELL, utf-8 and fossil command-line commit message

[Wayne Harris via Users list for the GNU Emacs text editor]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 01 Oct 2022 16:09:41 -0300
>> From:  Wayne Harris via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>
>> 
>> > Maybe you have other customizations that cause this.  What does the
>> > below say:
>> >
>> >   M-: default-process-coding-system RET
>> 
>> It says 
>> 
>> %(print default-process-coding-system)
>> (utf-8-dos . utf-8-unix)
>> %
>> 
>> I tried to change it to just cp1252.
>> 
>> --8<---------------cut here---------------start------------->8---
>> %(setq default-process-coding-system (cons 'cp1252 'cp1252))
>
> Too late, I think.
>
> So I think the problem is that your customizations set up UTF-8
> everywhere, and that causes the problems.  My suggestion is to start
> from "emacs -Q", and if these commands work there, review your
> customizations until you find those which get in the way.
>
> If even "emacs -Q" doesn't work, I suggest to submit a bug report with
> all the details.

Okay.  We get the same result with ``emacs -Q''.

c:/my/path $ alias fs 'fossil $*'
c:/my/path $ echo kkk >> encoding.txt 
c:/my/path $ fs changes
EDITED     encoding.txt

c:/my/path $ (print default-process-coding-system)
(undecided-dos . undecided-unix)

c:/my/path $ (or buffer-file-coding-system "it is nil")
it is nil

c:/my/path $ fs commit -m 'Naiveté'
[...]
Sync done, wire bytes sent: 3234  received: 309  ip: 5.161.138.46

c:/my/path $ fs timeline -n 1
=== 2022-10-02 ===
13:11:20 [febbbf0441] *CURRENT* Naiveté (user: mer tags: trunk)
--- entry limit (1) reached ---
c:/my/path $ 

I then tried to set the ESHELL buffer to utf-8, but I think the encoding
is mangled on the way to the server, so when it comes back it's already
too late.  (No change when viewing the timeline.)

> In general, in core we don't pass arbitrary text via command-line
> arguments on MS-Windows; instead, we write text to a temporary file
> and ask the program to read text from there -- for this very reason.
> Most VCS commands accept -f or -F switch telling them to read the log
> message from a file; I suggest to use that instead of fighting the
> UTF-8 uphill battle.

I'll not do commits on the command-line anymore.

>> %fs timeline -n 1
>> === 2022-10-01 ===
>> 19:04:34 [d992644b4b] *CURRENT* Naiveté. (user: mer tags: trunk)
>> --- entry limit (1) reached ---
>
> Could it be that 'fs' expects UTF-8?

I don't know.  I have only a few weeks of experience with it.  By the
way, ``fs'' is my alias for ``fossil'' (https://fossil-scm.org/).

> Is that a native Windows port or a Cygwin port?

It looks native to me.

%file c:/my/path/to/bin/fossil.exe
c:/my/path/to/bin/fossil.exe: PE32+ executable (console) x86-64, for MS Windows

%ldd c:/my/path/to/bin/fossil.exe
	ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffbe0ef0000)
	KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ffbe0c30000)
	KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ffbde940000)
	WS2_32.dll => /c/WINDOWS/System32/WS2_32.dll (0x7ffbe0140000)
	RPCRT4.dll => /c/WINDOWS/System32/RPCRT4.dll (0x7ffbdf610000)
	ADVAPI32.dll => /c/WINDOWS/System32/ADVAPI32.dll (0x7ffbe0820000)
	msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ffbdeff0000)
	sechost.dll => /c/WINDOWS/System32/sechost.dll (0x7ffbe0cf0000)
	USER32.dll => /c/WINDOWS/System32/USER32.dll (0x7ffbe0270000)
	win32u.dll => /c/WINDOWS/System32/win32u.dll (0x7ffbde6f0000)
	GDI32.dll => /c/WINDOWS/System32/GDI32.dll (0x7ffbdf5e0000)
	DNSAPI.dll => /c/WINDOWS/SYSTEM32/DNSAPI.dll (0x7ffbddaa0000)
	gdi32full.dll => /c/WINDOWS/System32/gdi32full.dll (0x7ffbdecf0000)
	msvcp_win.dll => /c/WINDOWS/System32/msvcp_win.dll (0x7ffbde720000)
	ucrtbase.dll => /c/WINDOWS/System32/ucrtbase.dll (0x7ffbde7c0000)
	CRYPT32.dll => /c/WINDOWS/System32/CRYPT32.dll (0x7ffbdee00000)
	bcrypt.dll => /c/WINDOWS/System32/bcrypt.dll (0x7ffbdecc0000)

> The latter will probably expect UTF-8 on the command line, and you
> cannot give it that.

It seems to me that Windows is mangling the command-line bytes.  Why
does it do that?  Is it trying to be helpful?