msys2 build path problems + copy-paste english results in chinese characters

msys2 build path problems + copy-paste english results in chinese characters

[arthur miller]

[-- Attachment #1.1: Type: text/plain, Size: 336 bytes --]

I copy pasted english text in newly compiled Emacs, and get back some text in Chinese (at least I think it is) 🙂 👍

As much as exciting that seems, I would still prefer to get back text in English.

Also, as seen exec-path is wrong. I started, as recommended, via windows means (shortcuts) instead of msys/mingw prompts.



[-- Attachment #1.2: Type: text/html, Size: 1627 bytes --]

[-- Attachment #2: Clip1.png --]
[-- Type: image/png, Size: 119019 bytes --]

Re: msys2 build path problems + copy-paste english results in chinese characters

[Lars Ingebrigtsen]

arthur miller <arthur.miller@live.com> writes:

> I copy pasted english text in newly compiled Emacs, and get back some text in
> Chinese (at least I think it is) 🙂 👍

What's your Emacs version (and if this is on master, what's the commit
you're at)?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: msys2 build path problems + copy-paste english results in chinese characters

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> arthur miller <arthur.miller@live.com> writes:
>
>> I copy pasted english text in newly compiled Emacs, and get back some text in
>> Chinese (at least I think it is) 🙂 👍
>
> What's your Emacs version (and if this is on master, what's the commit
> you're at)?

I installed msys2 clean + pulled Emacs like an hour/half hour before I did that
screenshot. I couldn't get emacsclient to work after rebuild, so I was checking
my paths and entire setup. Client/server works with emacs -Q, but suddenly my
config does not seem to work, and I used to have it working without problems
since back in winxp.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Lars Ingebrigtsen]

Arthur Miller <arthur.miller@live.com> writes:

> I installed msys2 clean + pulled Emacs like an hour/half hour before I
> did that screenshot.

`M-x report-emacs-bug' should give you the revision.  It'll look like
this:

Repository revision: 6a60bd475d67b7e8c9184836abf7eea229183066


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: msys2 build path problems + copy-paste english results in chinese characters

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> I installed msys2 clean + pulled Emacs like an hour/half hour before I
>> did that screenshot.
>
> `M-x report-emacs-bug' should give you the revision.  It'll look like
> this:
>
> Repository revision: 6a60bd475d67b7e8c9184836abf7eea229183066

Indeed, didn't thought of it. Anyway, I have booted into Arch anyway. I think I
know the problem; I pasted with the middle mouse button; does not work in Windows.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Eli Zaretskii]

> From: arthur miller <arthur.miller@live.com>
> Date: Wed, 1 Dec 2021 15:04:50 +0000
> 
> I copy pasted english text in newly compiled Emacs, and get back some text in Chinese (at least I think it is)

Copy-pasted from which application?

If it's an MSYS2 application, then I suspect the problem is MSYS2 apps
encode copied text in UTF-8 (since Cygwin does, AFAIK), whereas a
native MS-Windows build of Emacs expects a UTF-16 encoding.

> Also, as seen exec-path is wrong. I started, as recommended, via windows means (shortcuts) instead of
> msys/mingw prompts.

Wrong how?  I don't see anything about exec-path in the image you
posted.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: arthur miller <arthur.miller@live.com>
>> Date: Wed, 1 Dec 2021 15:04:50 +0000
>> 
>> I copy pasted english text in newly compiled Emacs, and get back some text in Chinese (at least I think it is)
>
> Copy-pasted from which application?
Emacs!

I copy-pasted the raw above, I did it with the mouse though. :-) After I took the
screenshot I redo it by marking and copying via M-w and C-y and it worked.

> If it's an MSYS2 application, then I suspect the problem is MSYS2 apps
> encode copied text in UTF-8 (since Cygwin does, AFAIK), whereas a
> native MS-Windows build of Emacs expects a UTF-16 encoding.
>
>> Also, as seen exec-path is wrong. I started, as recommended, via windows means (shortcuts) instead of
>> msys/mingw prompts.
>
> Wrong how?  I don't see anything about exec-path in the image you
> posted.

Look at warning from the native-comp in window below; it can not find assemblern
(gnu as). When looking at exec-path I see no paths from mingw present anywhere,
but I did found "." in the path, which I haven't put there myself.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 01 Dec 2021 19:25:54 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: arthur miller <arthur.miller@live.com>
> >> Date: Wed, 1 Dec 2021 15:04:50 +0000
> >> 
> >> I copy pasted english text in newly compiled Emacs, and get back some text in Chinese (at least I think it is)
> >
> > Copy-pasted from which application?
> Emacs!

You copied from Emacs to the same Emacs, and you get garbled text?

> I copy-pasted the raw above, I did it with the mouse though. :-)

So you marked the text with the mouse, and then did what? M-w? or
something else?  You must describe what you did exactly, because there
are too many unknown factors here.

> After I took the screenshot I redo it by marking and copying via M-w
> and C-y and it worked.

So M-w followed by C-y works.  What doesn't work?

> >> Also, as seen exec-path is wrong. I started, as recommended, via windows means (shortcuts) instead of
> >> msys/mingw prompts.
> >
> > Wrong how?  I don't see anything about exec-path in the image you
> > posted.
> 
> Look at warning from the native-comp in window below; it can not find assemblern
> (gnu as). When looking at exec-path I see no paths from mingw present anywhere,
> but I did found "." in the path, which I haven't put there myself.

The "." part is added by the MSYS2 Bash.  but I still don't understand
why it gets in the way.  Does the directory where you have gas.exe
appear on the system-wide PATH?

Re: msys2 build path problems + copy-paste english results in chinese characters

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 01 Dec 2021 19:25:54 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: arthur miller <arthur.miller@live.com>
>> >> Date: Wed, 1 Dec 2021 15:04:50 +0000
>> >> 
>> >> I copy pasted english text in newly compiled Emacs, and get back some text in Chinese (at least I think it is)
>> >
>> > Copy-pasted from which application?
>> Emacs!
>
> You copied from Emacs to the same Emacs, and you get garbled text?
>
>> I copy-pasted the raw above, I did it with the mouse though. :-)
>
> So you marked the text with the mouse, and then did what? M-w? or
> something else?  You must describe what you did exactly, because there
> are too many unknown factors here.
>
>> After I took the screenshot I redo it by marking and copying via M-w
>> and C-y and it worked.
>
> So M-w followed by C-y works.  What doesn't work?
I pasted it with the middle mouse button; so now when you asked, I should
probably say oups :-). It was many months of X11 and very few sessions in win32
environment.

>> >> Also, as seen exec-path is wrong. I started, as recommended, via windows means (shortcuts) instead of
>> >> msys/mingw prompts.
>> >
>> > Wrong how?  I don't see anything about exec-path in the image you
>> > posted.
>> 
>> Look at warning from the native-comp in window below; it can not find assemblern
>> (gnu as). When looking at exec-path I see no paths from mingw present anywhere,
>> but I did found "." in the path, which I haven't put there myself.
>
> The "." part is added by the MSYS2 Bash.  but I still don't understand
> why it gets in the way.  Does the directory where you have gas.exe
It is not considered very safe to have it in the path, so I am very suspicisious
to that.

>                          Does the directory where you have gas.exe
> appear on the system-wide PATH?

Nope; I haven't manually added any of msys paths to the system, I thought the
build would add some default paths to msys dirs.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Óscar Fuentes]

Arthur Miller <arthur.miller@live.com> writes:

>>> >> Also, as seen exec-path is wrong. I started, as recommended, via windows means (shortcuts) instead of
>>> >> msys/mingw prompts.
>>> >
>>> > Wrong how?  I don't see anything about exec-path in the image you
>>> > posted.
>>> 
>>> Look at warning from the native-comp in window below; it can not find assemblern
>>> (gnu as). When looking at exec-path I see no paths from mingw present anywhere,
>>> but I did found "." in the path, which I haven't put there myself.
>>
>> The "." part is added by the MSYS2 Bash.  but I still don't understand
>> why it gets in the way.  Does the directory where you have gas.exe
> It is not considered very safe to have it in the path, so I am very suspicisious
> to that.
>
>>                          Does the directory where you have gas.exe
>> appear on the system-wide PATH?
>
> Nope; I haven't manually added any of msys paths to the system, I thought the
> build would add some default paths to msys dirs.

Try this in your .emacs :

(let ((dir (file-name-directory (car command-line-args))))
  (setenv "PATH" (concat (getenv "PATH") path-separator dir))
  (setq exec-path (append exec-path (list dir))))

Re: msys2 build path problems + copy-paste english results in chinese characters

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 02 Dec 2021 00:01:41 +0100
> 
> >>                          Does the directory where you have gas.exe
> >> appear on the system-wide PATH?
> >
> > Nope; I haven't manually added any of msys paths to the system, I thought the
> > build would add some default paths to msys dirs.
> 
> Try this in your .emacs :
> 
> (let ((dir (file-name-directory (car command-line-args))))
>   (setenv "PATH" (concat (getenv "PATH") path-separator dir))
>   (setq exec-path (append exec-path (list dir))))

Changing PATH from within Emacs is not recommended, it will bite you
down the road when you least expect it.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> >>                          Does the directory where you have gas.exe
>> >> appear on the system-wide PATH?
>> >
>> > Nope; I haven't manually added any of msys paths to the system, I thought the
>> > build would add some default paths to msys dirs.
>> 
>> Try this in your .emacs :
>> 
>> (let ((dir (file-name-directory (car command-line-args))))
>>   (setenv "PATH" (concat (getenv "PATH") path-separator dir))
>>   (setq exec-path (append exec-path (list dir))))
>
> Changing PATH from within Emacs is not recommended, it will bite you
> down the road when you least expect it.

I'm using that since many years ago without problems. It makes possible
to run all the other Mingw-w64 executables installed on the same
directory as emacs.exe without changing the global PATH or using a
script to start Emacs.

If you know a better approach and/or wish to describe a failure mode for
the above code...

Re: msys2 build path problems + copy-paste english results in chinese characters

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 02 Dec 2021 10:42:49 +0100
> 
> >> (let ((dir (file-name-directory (car command-line-args))))
> >>   (setenv "PATH" (concat (getenv "PATH") path-separator dir))
> >>   (setq exec-path (append exec-path (list dir))))
> >
> > Changing PATH from within Emacs is not recommended, it will bite you
> > down the road when you least expect it.
> 
> I'm using that since many years ago without problems.

Sheer luck.

> If you know a better approach

Yes, change PATH outside of Emacs.  Then live happily ever after.

> and/or wish to describe a failure mode for the above code...

One failure is that you put directories with forward slashes into the
environment of the programs you invoke, and not all of them like that
(although most do cope with that).

Another problem is that after this, PATH used by Emacs and PATH used
by sub-processes is different.

Yet another problem, specific to invoking MSYS2 commands, is that the
directory might be incorrectly encoded (if it includes non-ASCII
characters), since MSYS2 programs expect UTF-8 encoding AFAIK, whereas
Emacs encodes it using the system codepage.

There might be other problems, I'm not sure I remember all of them.  I
just learned long ago not to do that.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Thu, 02 Dec 2021 10:42:49 +0100
>> 
>> >> (let ((dir (file-name-directory (car command-line-args))))
>> >>   (setenv "PATH" (concat (getenv "PATH") path-separator dir))
>> >>   (setq exec-path (append exec-path (list dir))))
>> >
>> > Changing PATH from within Emacs is not recommended, it will bite you
>> > down the road when you least expect it.
>> 
>> I'm using that since many years ago without problems.
>
> Sheer luck.
>
>> If you know a better approach
>
> Yes, change PATH outside of Emacs.  Then live happily ever after.

That's precisely what I prefer to avoid.

>> and/or wish to describe a failure mode for the above code...
>
> One failure is that you put directories with forward slashes into the
> environment of the programs you invoke, and not all of them like that
> (although most do cope with that).

Maybe if an application parses PATH on a broken way. So far I found
none.

> Another problem is that after this, PATH used by Emacs and PATH used
> by sub-processes is different.

I don't know how this could be a problem, even less when emacs.exe lives
in the directory added to PATH.

> Yet another problem, specific to invoking MSYS2 commands, is that the
> directory might be incorrectly encoded (if it includes non-ASCII
> characters), since MSYS2 programs expect UTF-8 encoding AFAIK, whereas
> Emacs encodes it using the system codepage.

Well, adding directories containing MSYS2/Cygwin applications to PATH is
risky, something to avoid. Fortunately, on a MSYS2 setup MSYS2 and
mingw-w64 binaries are strictly separated.

> There might be other problems, I'm not sure I remember all of them.  I
> just learned long ago not to do that.

My experience says otherwise, and having to set up PATH so Emacs can
execute the applications installed on its same directory is not trivial
for inexperienced users and an annoyance for the rest. IMO
runemacs.exe/emacsclient.exe should add their directory to PATH before
invoking emacs.exe.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 02 Dec 2021 16:40:43 +0100
> 
> >> If you know a better approach
> >
> > Yes, change PATH outside of Emacs.  Then live happily ever after.
> 
> That's precisely what I prefer to avoid.

There's no need to avoid it.

> > One failure is that you put directories with forward slashes into the
> > environment of the programs you invoke, and not all of them like that
> > (although most do cope with that).
> 
> Maybe if an application parses PATH on a broken way. So far I found
> none.

I've definitely seen a few in the past.

> > Another problem is that after this, PATH used by Emacs and PATH used
> > by sub-processes is different.
> 
> I don't know how this could be a problem, even less when emacs.exe lives
> in the directory added to PATH.

It could be a problem because sub-processes will be able to find
programs that Emacs might not find.

> > Yet another problem, specific to invoking MSYS2 commands, is that the
> > directory might be incorrectly encoded (if it includes non-ASCII
> > characters), since MSYS2 programs expect UTF-8 encoding AFAIK, whereas
> > Emacs encodes it using the system codepage.
> 
> Well, adding directories containing MSYS2/Cygwin applications to PATH is
> risky, something to avoid. Fortunately, on a MSYS2 setup MSYS2 and
> mingw-w64 binaries are strictly separated.

But the OP definitely wanted MSYS2 executables, that's why he invoked
Emacs from Bash (which is an MSYS2 program).

Re: msys2 build path problems + copy-paste english results in chinese characters

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Thu, 02 Dec 2021 16:40:43 +0100
>> 
>> >> If you know a better approach
>> >
>> > Yes, change PATH outside of Emacs.  Then live happily ever after.
>> 
>> That's precisely what I prefer to avoid.
>
> There's no need to avoid it.
>
>> > One failure is that you put directories with forward slashes into the
>> > environment of the programs you invoke, and not all of them like that
>> > (although most do cope with that).
>> 
>> Maybe if an application parses PATH on a broken way. So far I found
>> none.
>
> I've definitely seen a few in the past.
>
>> > Another problem is that after this, PATH used by Emacs and PATH used
>> > by sub-processes is different.
>> 
>> I don't know how this could be a problem, even less when emacs.exe lives
>> in the directory added to PATH.
>
> It could be a problem because sub-processes will be able to find
> programs that Emacs might not find.
>
>> > Yet another problem, specific to invoking MSYS2 commands, is that the
>> > directory might be incorrectly encoded (if it includes non-ASCII
>> > characters), since MSYS2 programs expect UTF-8 encoding AFAIK, whereas
>> > Emacs encodes it using the system codepage.
>> 
>> Well, adding directories containing MSYS2/Cygwin applications to PATH is
>> risky, something to avoid. Fortunately, on a MSYS2 setup MSYS2 and
>> mingw-w64 binaries are strictly separated.

Not just strictly separated but they don't seem to like to be mixed. Now
coreutils get installed into "msys" (/usr/bin) while GCC gets installed into
mingw64 (/mingw64/bin). So if I wish emacs to find gcc, as & co, and not run it
from the command line, I have to add mingw64/bin to my OS path. If I also wish
Emacs to find gnu ls, I have to add /usr/bin to OS path to, but they are not
recommended to mix, and you also don't recommend setting path from within Emacs
either, so I am a little bit puzzled here what would is a good practice.

I have currently added mingw first, and msys after to the OS paths, so gcc & co
will find it's binaries first, how well it will work I will see.

And I also discovered I can't send mail, authentication failed, which is 
probably .authinfo file not being found or not being decrypted, which also used
to work before. I don't know yet if it is also related to paths or something
else; I am not into tweaking my system more today, so that will be another day.

MSYS2 PATH problems with native compilation (was: msys2 build path problems + copy-paste english results in chinese characters)

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Thu, 02 Dec 2021 00:01:41 +0100
>> 
>> >>                          Does the directory where you have gas.exe
>> >> appear on the system-wide PATH?
>> >
>> > Nope; I haven't manually added any of msys paths to the system, I thought the
>> > build would add some default paths to msys dirs.
>> 
>> Try this in your .emacs :
>> 
>> (let ((dir (file-name-directory (car command-line-args))))
>>   (setenv "PATH" (concat (getenv "PATH") path-separator dir))
>>   (setq exec-path (append exec-path (list dir))))
>
> Changing PATH from within Emacs is not recommended, it will bite you
> down the road when you least expect it.

Revisiting this...

I just checked in the recipe for building Emacs 28.0.90 on MinGW-w64
(MSYS2) with native compilation enabled and found a serious problem.

The Emacs MinGW-w64/MSYS2 package now depends on libgccjit package,
which means that libgccjit will be present when Emacs is installed. But
if Emacs runs without its bin/ directory in PATH, libgccjit is not
functional (an error about missing as.exe is shown.)

Creating a desktop icon for runemacs.exe and starting Emacs from there
is common. Also for MSYS2 users is common to work with multiple
architectures (mingw, clang, ucrt with their 32/64 bits variants)
simultaneously, and putting the binaries of one of those architectures
in global PATH is problematic.

Hence it would be very convenient to use libgccjit without touching PATH.

I ask again: would it be ok to add emacs.exe directory to PATH from
runemacs.exe or emacs.exe itself?

Set PATH for the emacs instances used for generating the .eln files?

Other solution?

Re: MSYS2 PATH problems with native compilation (was: msys2 build path problems + copy-paste english results in chinese characters)

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 06 Dec 2021 01:38:14 +0100
> 
> I just checked in the recipe for building Emacs 28.0.90 on MinGW-w64
> (MSYS2) with native compilation enabled and found a serious problem.
> 
> The Emacs MinGW-w64/MSYS2 package now depends on libgccjit package,
> which means that libgccjit will be present when Emacs is installed.

What do you mean by "depends on libgccjit"?  Are you linking Emacs
statically against the libgccjit DLL (via the import library)?  If
not, Emacs on Windows loads libgccjit dynamically, when it is first
requested, and it doesn't need that library to start.  So
theoretically, MinGW64 users could decide not to install libgccjit, or
have it unavailable to Emacs, and they will still be able to run
Emacs, just not to natively-compile *.el files that you didn't compile
ahead of time and provided in the binary distribution.

> But if Emacs runs without its bin/ directory in PATH, libgccjit is
> not functional (an error about missing as.exe is shown.)

Only if you try to natively-compile *.el files (or Emacs tries doing
that in the background).  And, as you point out, not only libgccjit is
needed if you do want to compile, the assembler and the linker are
also needed.  So the PATH problem is not only about libgccjit.

> Creating a desktop icon for runemacs.exe and starting Emacs from there
> is common. Also for MSYS2 users is common to work with multiple
> architectures (mingw, clang, ucrt with their 32/64 bits variants)
> simultaneously, and putting the binaries of one of those architectures
> in global PATH is problematic.
> 
> Hence it would be very convenient to use libgccjit without touching PATH.

How is libgccjit different from any other DLL that Emacs loads
dynamically, like libpng, for example?  How do your users, who work
with multiple architectures, cope with that wrt other DLLs that Emacs
uses?

> I ask again: would it be ok to add emacs.exe directory to PATH from
> runemacs.exe or emacs.exe itself?
> 
> Set PATH for the emacs instances used for generating the .eln files?

"OK" from what POV?  The MSYS2 project makes its own rules for
configuring binary distributions, and those decisions are not
necessarily aligned with the upstream project.  Why do you ask here
whether something you'd like to do in your distribution is OK or not?
If you want to know my opinions, you already heard them.  (And I still
don't think I understand the underlying problem, even after your
additional explanations, see above.)

> Other solution?

If you cannot put a DLL on PATH, the usual solution is to have it in
the same directory as the .exe which needs it.  Would that be a
satisfactory solution for your problems?

Re: MSYS2 PATH problems with native compilation

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> I just checked in the recipe for building Emacs 28.0.90 on MinGW-w64
>> (MSYS2) with native compilation enabled and found a serious problem.
>> 
>> The Emacs MinGW-w64/MSYS2 package now depends on libgccjit package,
>> which means that libgccjit will be present when Emacs is installed.
>
> What do you mean by "depends on libgccjit"?

It is a package dependency, meaning that if the user installs the emacs
package, it is guaranteed that libgccjit will be installed too.

>> Creating a desktop icon for runemacs.exe and starting Emacs from there
>> is common. Also for MSYS2 users is common to work with multiple
>> architectures (mingw, clang, ucrt with their 32/64 bits variants)
>> simultaneously, and putting the binaries of one of those architectures
>> in global PATH is problematic.
>> 
>> Hence it would be very convenient to use libgccjit without touching PATH.
>
> How is libgccjit different from any other DLL that Emacs loads
> dynamically, like libpng, for example?  How do your users, who work
> with multiple architectures, cope with that wrt other DLLs that Emacs
> uses?

DLLs that Emacs load dynamically are not a problem, they all work fine
(including libgccjit dll). The problem is for executables invoked by
Emacs, because Emacs searchs for them using exec-path (which itself
depends on the initial value of PATH.) This is different to what
CreateProcess does which, as you very well know, always searches the
executable on the same directory as the calling process' executable,
first thing. This means that Emacs deviates from the stablished rule on
Windows, and that deviation causes a degraded experience.

To recap: libgccjit dll is fine, but as.exe/ld.exe/etcetera are not found
despite being installed right there along emacs.exe.

>> I ask again: would it be ok to add emacs.exe directory to PATH from
>> runemacs.exe or emacs.exe itself?
>> 
>> Set PATH for the emacs instances used for generating the .eln files?
>
> "OK" from what POV?

From your POV, of course. I'm talking about making changes to Emacs.

> The MSYS2 project makes its own rules for
> configuring binary distributions, and those decisions are not
> necessarily aligned with the upstream project.  Why do you ask here
> whether something you'd like to do in your distribution is OK or not?

We could locally patch Emacs, of course, but "fixing" the upstream
project is preferable.

> If you want to know my opinions, you already heard them.  (And I still
> don't think I understand the underlying problem, even after your
> additional explanations, see above.)
>
>> Other solution?
>
> If you cannot put a DLL on PATH, the usual solution is to have it in
> the same directory as the .exe which needs it.  Would that be a
> satisfactory solution for your problems?

This is not about DLLs, as stated above.

The general problem is that Emacs is installed along all the tools it
needs, but they are invisible to Emacs unless you explicitly say "they
are right there where you are!". That's not the experience we get on
GNU/Linux and other systems, where Emacs has no problem finding any
executable as long as it is installed where the package manager puts it
by default.

Re: MSYS2 PATH problems with native compilation

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 06 Dec 2021 23:48:28 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I just checked in the recipe for building Emacs 28.0.90 on MinGW-w64
> >> (MSYS2) with native compilation enabled and found a serious problem.
> >> 
> >> The Emacs MinGW-w64/MSYS2 package now depends on libgccjit package,
> >> which means that libgccjit will be present when Emacs is installed.
> >
> > What do you mean by "depends on libgccjit"?
> 
> It is a package dependency, meaning that if the user installs the emacs
> package, it is guaranteed that libgccjit will be installed too.
> 
> >> Creating a desktop icon for runemacs.exe and starting Emacs from there
> >> is common. Also for MSYS2 users is common to work with multiple
> >> architectures (mingw, clang, ucrt with their 32/64 bits variants)
> >> simultaneously, and putting the binaries of one of those architectures
> >> in global PATH is problematic.
> >> 
> >> Hence it would be very convenient to use libgccjit without touching PATH.
> >
> > How is libgccjit different from any other DLL that Emacs loads
> > dynamically, like libpng, for example?  How do your users, who work
> > with multiple architectures, cope with that wrt other DLLs that Emacs
> > uses?
> 
> DLLs that Emacs load dynamically are not a problem, they all work fine
> (including libgccjit dll). The problem is for executables invoked by
> Emacs, because Emacs searchs for them using exec-path (which itself
> depends on the initial value of PATH.) This is different to what
> CreateProcess does which, as you very well know, always searches the
> executable on the same directory as the calling process' executable,
> first thing. This means that Emacs deviates from the stablished rule on
> Windows, and that deviation causes a degraded experience.
> 
> To recap: libgccjit dll is fine, but as.exe/ld.exe/etcetera are not found
> despite being installed right there along emacs.exe.

So you were talking about libgccjit, but actually meant the assembler
and the linker programs, not the libgccjit DLL?  That was hard to
guess.

If the problem is with the Binutils that are invoked as part of the
compilation, then I think I understand the problem even less.  For
starters, AFAIK we don't search for those in the Emacs code, it's
libgccjit that invokes them in its own code, according to the rules of
a "usual" GCC installation.  Andrea, am I mistaken?  I don't see any
use of exec-path in the nativecomp code that would look for the
assembler and the linker, am I missing something?

If I'm right, then nothing you do with PATH in Emacs can guarantee
that native-compilation will find Binutils, because that depends on
how GCC and Binutils were installed and how they search for the
programs they need.  IOW, I believe that libgccjit relies on a
functional GCC installation, including the subprograms under libexec
and (perhaps) some system header files?  I don't even know what it
expects, because AFAIU its prerequisite is a working GCC and Binutils
installation.

But if PATH does make a differenceand can solve your problem, by some
indirect way, then did you try to put the assembler and the linker in
the libexec/emacs/ tree, where we keep auxiliary programs Emacs
invokes?  Emacs searches there by default.

> >> I ask again: would it be ok to add emacs.exe directory to PATH from
> >> runemacs.exe or emacs.exe itself?
> >> 
> >> Set PATH for the emacs instances used for generating the .eln files?
> >
> > "OK" from what POV?
> 
> >From your POV, of course. I'm talking about making changes to Emacs.

You want Emacs to change the user's PATH by default?  That'd be
unthinkable, IMO.  Programs have no business messing with user's PATH,
since that could easily mess up user's system.  E.g., suppose the user
already has a development environment installed and on PATH: then you
are silently causing "M-x compile" to use different Binutils programs
than what the user intended.

> The general problem is that Emacs is installed along all the tools it
> needs, but they are invisible to Emacs unless you explicitly say "they
> are right there where you are!". That's not the experience we get on
> GNU/Linux and other systems, where Emacs has no problem finding any
> executable as long as it is installed where the package manager puts it
> by default.

I don't know what kind of GNU/Linux installations you have in mind,
but I don't know about any reliable installation technique that causes
all the installed tools to seamlessly work together, except by either
modifying PATH or installing all the tools in the same common tree.
Anything else is much more complex and fragile.  Having several
incompatible development environments on the same machine active
simultaneously is IME a very rare use case, but if your users do that
frequently, and you don't have some system of batch files/scripts to
easily and reliably switch between the environments, then your users
are in for a lot of trouble.

Re: MSYS2 PATH problems with native compilation

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> To recap: libgccjit dll is fine, but as.exe/ld.exe/etcetera are not found
>> despite being installed right there along emacs.exe.
>
> So you were talking about libgccjit, but actually meant the assembler
> and the linker programs, not the libgccjit DLL?  That was hard to
> guess.

As far as native-comp is concerned, the required feature is the presence
of a functional libgccjit install. In my original message I never
mentioned a dll.

> If the problem is with the Binutils that are invoked as part of the
> compilation, then I think I understand the problem even less.  For
> starters, AFAIK we don't search for those in the Emacs code, it's
> libgccjit that invokes them in its own code, according to the rules of
> a "usual" GCC installation.  Andrea, am I mistaken?  I don't see any
> use of exec-path in the nativecomp code that would look for the
> assembler and the linker, am I missing something?

libgccjit probably uses `system` or some other shell-related mechanism
for invoking the assembler and linker.

> If I'm right, then nothing you do with PATH in Emacs can guarantee
> that native-compilation will find Binutils, because that depends on
> how GCC and Binutils were installed and how they search for the
> programs they need.

Having in PATH the /bin directory where Emacs and gcc are installed
solves the problem.

> IOW, I believe that libgccjit relies on a
> functional GCC installation, including the subprograms under libexec
> and (perhaps) some system header files?  I don't even know what it
> expects, because AFAIU its prerequisite is a working GCC and Binutils
> installation.
>
> But if PATH does make a differenceand can solve your problem, by some
> indirect way, then did you try to put the assembler and the linker in
> the libexec/emacs/ tree, where we keep auxiliary programs Emacs
> invokes?  Emacs searches there by default.

Duplicating executables here and there just to appease Emacs is a not
very appealing, to say the least. Furthermore, those executables might
need other files present, that scenario is best described as "can of
worms".

>> >> I ask again: would it be ok to add emacs.exe directory to PATH from
>> >> runemacs.exe or emacs.exe itself?
>> >> 
>> >> Set PATH for the emacs instances used for generating the .eln files?
>> >
>> > "OK" from what POV?
>> 
>> >From your POV, of course. I'm talking about making changes to Emacs.
>
> You want Emacs to change the user's PATH by default?

No, not necessarily. For solving the problem with libgccjit not finding
as/ld, what I want is to set PATH for the emacs.exe instance spawned
for compiling the .eln.

> That'd be unthinkable, IMO.

Currently the user is forced to add emacs.exe to PATH if he wants a
functional native-comp. Not to mention executing tools that are
installed on the same tree as Emacs in general.

> Programs have no business messing with user's PATH,
> since that could easily mess up user's system.  E.g., suppose the user
> already has a development environment installed and on PATH: then you
> are silently causing "M-x compile" to use different Binutils programs
> than what the user intended.

Not if emacs.exe directory is appended. However, I'm proposing to set
PATH transiently, see above.

>> The general problem is that Emacs is installed along all the tools it
>> needs, but they are invisible to Emacs unless you explicitly say "they
>> are right there where you are!". That's not the experience we get on
>> GNU/Linux and other systems, where Emacs has no problem finding any
>> executable as long as it is installed where the package manager puts it
>> by default.
>
> I don't know what kind of GNU/Linux installations you have in mind,
> but I don't know about any reliable installation technique that causes
> all the installed tools to seamlessly work together, except by either
> modifying PATH or installing all the tools in the same common tree.

Precisely, MSYS2/Mingw-w64 installs everything in the same tree.

> Anything else is much more complex and fragile.  Having several
> incompatible development environments on the same machine active
> simultaneously is IME a very rare use case,

In MSYS2/Mingw-w64 it is quite common.

> but if your users do that
> frequently, and you don't have some system of batch files/scripts to
> easily and reliably switch between the environments, then your users
> are in for a lot of trouble.

Well, that's something for the user to care about. Our job is to provide
an Emacs that works as installed, and right now native-comp is broken
unless the user sets PATH globally.

Re: MSYS2 PATH problems with native compilation

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 07 Dec 2021 17:09:45 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> To recap: libgccjit dll is fine, but as.exe/ld.exe/etcetera are not found
> >> despite being installed right there along emacs.exe.
> >
> > So you were talking about libgccjit, but actually meant the assembler
> > and the linker programs, not the libgccjit DLL?  That was hard to
> > guess.
> 
> As far as native-comp is concerned, the required feature is the presence
> of a functional libgccjit install. In my original message I never
> mentioned a dll.

libgccjit _is_ a DLL.  I understand now that you meant to name a
package instead, but expecting me to realize that, let alone know what
is the contents of that package, is a lot of expectations.  Please try
to be more explicit in the future, to make the discussion more
efficient.

> > If the problem is with the Binutils that are invoked as part of the
> > compilation, then I think I understand the problem even less.  For
> > starters, AFAIK we don't search for those in the Emacs code, it's
> > libgccjit that invokes them in its own code, according to the rules of
> > a "usual" GCC installation.  Andrea, am I mistaken?  I don't see any
> > use of exec-path in the nativecomp code that would look for the
> > assembler and the linker, am I missing something?
> 
> libgccjit probably uses `system` or some other shell-related mechanism
> for invoking the assembler and linker.

Maybe so, but even if it does, it doesn't necessarily rely on PATH to
find the executables.  For example (and I don't know if this is
pertinent), if it needs to invoke the cc1.exe C compiler, it will not
generally find it on PATH.

> Having in PATH the /bin directory where Emacs and gcc are installed
> solves the problem.
> 
> > IOW, I believe that libgccjit relies on a
> > functional GCC installation, including the subprograms under libexec
> > and (perhaps) some system header files?  I don't even know what it
> > expects, because AFAIU its prerequisite is a working GCC and Binutils
> > installation.
> >
> > But if PATH does make a differenceand can solve your problem, by some
> > indirect way, then did you try to put the assembler and the linker in
> > the libexec/emacs/ tree, where we keep auxiliary programs Emacs
> > invokes?  Emacs searches there by default.
> 
> Duplicating executables here and there just to appease Emacs is a not
> very appealing, to say the least.

But above you say that placing these executables in the same directory
as Emacs solves the problem, so you are going to duplicate them
anyway, right?  Because the "normal" GCC installation doesn't
necessarily put them in the same directory, right?  And if they _are_
installed in the same directory, then what is again the problem of
adding that single directory to the system-wide PATH?

> > You want Emacs to change the user's PATH by default?
> 
> No, not necessarily. For solving the problem with libgccjit not finding
> as/ld, what I want is to set PATH for the emacs.exe instance spawned
> for compiling the .eln.

But that solves only part of the problem, because the compilation is
not necessarily in a subprocess.  Some of the comp.el commands use the
same process to compile the files.  So with your proposal, some
commands will maybe work, but others won't.

> > That'd be unthinkable, IMO.
> 
> Currently the user is forced to add emacs.exe to PATH if he wants a
> functional native-comp. Not to mention executing tools that are
> installed on the same tree as Emacs in general.

That's the user's responsibility and his/her action.  Only the user
knows (or can know) how to add a directory correctly to PATH without
messing up his/her system configuration.

> > Programs have no business messing with user's PATH,
> > since that could easily mess up user's system.  E.g., suppose the user
> > already has a development environment installed and on PATH: then you
> > are silently causing "M-x compile" to use different Binutils programs
> > than what the user intended.
> 
> Not if emacs.exe directory is appended.

If you append it, then there could be a different as.exe/ld.exe
earlier on PATH, and you have the same problem.

> > I don't know what kind of GNU/Linux installations you have in mind,
> > but I don't know about any reliable installation technique that causes
> > all the installed tools to seamlessly work together, except by either
> > modifying PATH or installing all the tools in the same common tree.
> 
> Precisely, MSYS2/Mingw-w64 installs everything in the same tree.

So why on Earth not tell your users to add that single bin/ directory
to system-wide PATH, and be done with that?

> Well, that's something for the user to care about. Our job is to provide
> an Emacs that works as installed, and right now native-comp is broken
> unless the user sets PATH globally.

Then I suggest to tell the users to set PATH globally.  That's the
correct solution, on any system, because it makes the MinGW64 programs
accessible from anywhere and by any program running on the system.

Re: MSYS2 PATH problems with native compilation

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Duplicating executables here and there just to appease Emacs is a not
>> very appealing, to say the least.
>
> But above you say that placing these executables in the same directory
> as Emacs solves the problem,

No, I didn't said that.

> so you are going to duplicate them
> anyway, right?  Because the "normal" GCC installation doesn't
> necessarily put them in the same directory, right?

gcc.exe, ld.exe, as.exe etc are on the same directory as emacs.exe. This
is no coincidence, is how packages are installed on MSYS2/Mingw-w64,
which follows a similar approach to GNU/Linux.

> And if they _are_
> installed in the same directory, then what is again the problem of
> adding that single directory to the system-wide PATH?

You say that adding PATH automatically from emacs is a no-no, but you
are fine with forcing the user to set PATH globally (thus affecting the
whole system) ??

>> > I don't know what kind of GNU/Linux installations you have in mind,
>> > but I don't know about any reliable installation technique that causes
>> > all the installed tools to seamlessly work together, except by either
>> > modifying PATH or installing all the tools in the same common tree.
>> 
>> Precisely, MSYS2/Mingw-w64 installs everything in the same tree.
>
> So why on Earth not tell your users to add that single bin/ directory
> to system-wide PATH, and be done with that?

I have no method for telling the users, apart from patching Emacs to
show a prominent message. Anyways, a package that requires further
action after installation (and changing the global environment, no less)
is anything but user-friendly.

>> Well, that's something for the user to care about. Our job is to provide
>> an Emacs that works as installed, and right now native-comp is broken
>> unless the user sets PATH globally.
>
> Then I suggest to tell the users to set PATH globally.  That's the
> correct solution, on any system, because it makes the MinGW64 programs
> accessible from anywhere and by any program running on the system.

The correct solution is to make native-comp work out of the box.
Transiently setting PATH while generating .eln is a fix. What do you
have against it?

Re: MSYS2 PATH problems with native compilation

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 07 Dec 2021 22:13:09 +0100
> 
> gcc.exe, ld.exe, as.exe etc are on the same directory as emacs.exe. This
> is no coincidence, is how packages are installed on MSYS2/Mingw-w64,
> which follows a similar approach to GNU/Linux.

So the only thing that's missing is that this directory is not on
PATH, and telling your users to add it to PATH would solve it cleanly
and easily.

(Btw, I don't believe libgccjit DLL is invoking gcc.exe, I think it
invokes cc1.exe directly.)

> > And if they _are_
> > installed in the same directory, then what is again the problem of
> > adding that single directory to the system-wide PATH?
> 
> You say that adding PATH automatically from emacs is a no-no, but you
> are fine with forcing the user to set PATH globally (thus affecting the
> whole system) ??

Of course!  Because the latter will be done by the user, who is
supposed to know his/her needs and the setup of his/her system, and is
therefore capable of changing PATH in ways that will not interfere
with whatever the user does on that system.  Whereas changing PATH
from a program which knows nothing about that stuff is likely to cause
trouble.

> > So why on Earth not tell your users to add that single bin/ directory
> > to system-wide PATH, and be done with that?
> 
> I have no method for telling the users, apart from patching Emacs to
> show a prominent message.

Aren't there MinGW64 installation instructions somewhere?
Instructions that aren't related to Emacs alone, but are general.
That's the place to tell them.  Seriously, installing programs and not
putting their directory on PATH is at least weird if not downright
wrong.

> Transiently setting PATH while generating .eln is a fix. What do you
> have against it?

I already told you: it will cause some uses of native-compilation
work, but others, the ones which don't start sub-processes, will
still fail.

And this is not an Emacs problem anyway, it's a MinGW64 problem:
MinGW64 installation instructions should tell users to add the MinGw64
bin/ directory to PATH.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 01 Dec 2021 23:39:23 +0100
> 
> > So M-w followed by C-y works.  What doesn't work?
> I pasted it with the middle mouse button; so now when you asked, I should
> probably say oups :-). It was many months of X11 and very few sessions in win32
> environment.

I don't think I understand the "oups" part: does it mean you figured
out what caused the problem?

> >> >> Also, as seen exec-path is wrong. I started, as recommended, via windows means (shortcuts) instead of
> >> >> msys/mingw prompts.
> >> >
> >> > Wrong how?  I don't see anything about exec-path in the image you
> >> > posted.
> >> 
> >> Look at warning from the native-comp in window below; it can not find assemblern
> >> (gnu as). When looking at exec-path I see no paths from mingw present anywhere,
> >> but I did found "." in the path, which I haven't put there myself.
> >
> > The "." part is added by the MSYS2 Bash.  but I still don't understand
> > why it gets in the way.  Does the directory where you have gas.exe
> It is not considered very safe to have it in the path, so I am very suspicisious
> to that.

MSYS2 does it for good reasons.  Since you invoked Emacs from the
MSYS2 Bash, something that is generally not recommended, you inherit
that, and have to deal with it.

> >                          Does the directory where you have gas.exe
> > appear on the system-wide PATH?
> 
> Nope; I haven't manually added any of msys paths to the system, I thought the
> build would add some default paths to msys dirs.

That's your problem: this is why Emacs started from Bash doesn't find
the assembler.  The assembler (and GCC/Binutils in general) are not
MSYS2 programs, they are native Windows programs.  Their directories
should be on PATH, so that you could invoke them from anywhere on your
system, not just from MSYS2 Bash command line.

Re: msys2 build path problems + copy-paste english results in chinese characters

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 05 Dec 2021 09:16:50 +0100
> 
> I have been back to windows now, and got server/dameon to work again, paths
> fixed etc. Then I have experienced something I think is a bug and wanted to
> test it in a clean Emacs, so I started new instance with -Q option. I copied
> text from my customized Emacs with by marking text and using kill-ring-save
> (M-w). In clean Emacs I pasted it with yank (C-y) and result came out
> completely scrambled, screenshot included.

I think the reason for this is in your customizations in Emacs from
which you did the M-w.  It sounds like it somehow uses an encoding
other than UTF-16 to encode the text it puts into the clipboard.

> Regarding what I think is bug; I can't make a frame from elisp if I pass in both
> width and height.

Can you please separate the bugs?  This is unrelated to the issue with
copy/paste.


> This code gives me "memory exhausted .." error (C stack
> overflow?):
> 
> (defvar emw--frame nil)
> 
> (let ((w (display-pixel-width))
>       (h (display-pixel-height)))
>         (setq emw--frame (make-frame 
>                           `((width . ,w)
>                             (height . ,h)
>                             (visibility . t)
>                             (auto-raise . nil)
>                             (skip-taskbar . t)
>                             (no-focus-on-map . t)
>                             (no-accept-focus . t)
>                             (undecorated . t)
>                             (unsplittable . t)
>                             (z-group . below)
>                             (no-other-frame . t)
>                             (minibuffer . nil)
>                             (tool-bar-lines . 0)
>                             (menu-bar-lines . 0)
>                             (left-fringe . 0)
>                             (right-fringe . 0)
>                             (border-width . 0)
>                             (internal-border-width . 0)
>                             (vertical-scroll-bars . nil)
>                             (horizontal-scroll-bars . nil)))))

When I evaluate the above in "emacs -Q", Emacs signals an error:

  (error "Value ‘below’ for z-group is not supported on Windows")

So I wonder why you get a different result.  What version of Emacs is
that?

Re: msys2 build path problems + copy-paste english results in chinese characters

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 05 Dec 2021 09:16:50 +0100
>> 
>> I have been back to windows now, and got server/dameon to work again, paths
>> fixed etc. Then I have experienced something I think is a bug and wanted to
>> test it in a clean Emacs, so I started new instance with -Q option. I copied
>> text from my customized Emacs with by marking text and using kill-ring-save
>> (M-w). In clean Emacs I pasted it with yank (C-y) and result came out
>> completely scrambled, screenshot included.
>
> I think the reason for this is in your customizations in Emacs from
> which you did the M-w.  It sounds like it somehow uses an encoding
> other than UTF-16 to encode the text it puts into the clipboard.

Ok. I'll investigate. This is same setup I had since quite some time, so it is a
bit strange, but Emacs has evolved. 

>> Regarding what I think is bug; I can't make a frame from elisp if I pass in both
>> width and height.
>
> Can you please separate the bugs?  This is unrelated to the issue with
> copy/paste.

I actually wanted to investigate what is going on and potentially write a bug
repport, but got all those other issues, couldn't even sent mail from gnus, so I
got back to Arch :).

>
>> This code gives me "memory exhausted .." error (C stack
>> overflow?):
>> 
>> (defvar emw--frame nil)
>> 
>> (let ((w (display-pixel-width))
>>       (h (display-pixel-height)))
>>         (setq emw--frame (make-frame 
>>                           `((width . ,w)
>>                             (height . ,h)
>>                             (visibility . t)
>>                             (auto-raise . nil)
>>                             (skip-taskbar . t)
>>                             (no-focus-on-map . t)
>>                             (no-accept-focus . t)
>>                             (undecorated . t)
>>                             (unsplittable . t)
>>                             (z-group . below)
>>                             (no-other-frame . t)
>>                             (minibuffer . nil)
>>                             (tool-bar-lines . 0)
>>                             (menu-bar-lines . 0)
>>                             (left-fringe . 0)
>>                             (right-fringe . 0)
>>                             (border-width . 0)
>>                             (internal-border-width . 0)
>>                             (vertical-scroll-bars . nil)
>>                             (horizontal-scroll-bars . nil)))))
>
> When I evaluate the above in "emacs -Q", Emacs signals an error:
>
>   (error "Value ‘below’ for z-group is not supported on Windows")
>
> So I wonder why you get a different result.  What version of Emacs is
> that?

As said, I get that error you saw on the screenshot. I didn't see that message
about z-group. It is a couple of days ago, when I started the thread.

I'll go back to windows later this evening and try to work out more what is
going on.